Posts tagged with corpora

Physicists talk about vibrational modes of a string, for example the superposition of sin(3x) and sin(5x). And the maths on a line segment or a disc can apply to electromagnetic fields, superstrings, sound waves, seasonal or multi-year or multi-century weather patterns, or the ODE’s for a particle wavefunction.

$E(x,y,z)= \sum_{k=1}^M {(a_k e^{(i \beta_k z)}+ b_k e^{(-i \beta_k z)})E_k(x,y)}$

$H(x,y,z)= \sum_{k=1}^M {(a_k e^{(i \beta_k z)}+ b_k e^{(-i \beta_k z)})H_k(x,y)}$

But if you take the perspective that an eigenbasis is constructed out of elements of a set, you can talk in the language of mathematics about a diverse range of topics that feel like “adding separable, independent pieces of little modes” together.

• paths that I walk to work
• mood patterns. Like a basis functioncould be
• that you always get more depressed in the winter
• and that is added in varying ratios to your workweek basis function, where Friday is the best
• and added to another “vibrational” mode which is the whips and wiggles of a particular client’s project
• If you want to talk about a non-scalar codomain, the things that I buy go through cycles. It’s autumn now and I am definitely going to be purchasing pumpkin-flavoured beer, coffee, pie, fuggineverythingIcangetapumpkinof.
• I usually smoke about 12 packs of cigarettes per year. However it’s not like I smoke one kretek every 2.53 days. Rather about once a month I hanker for a pack and then I do the whole pack within a day or two. So that’s a little “wavelet tooth” mode added onto the string of my purchasing behaviour, at irregular yet regular intervals. It’s not quite an addiction but it is a habit, or at least a repeated activity.
• Internet traffic patterns on a macro scale: at the holidays every epicyclic year people go home, get bored of their families, and surf Facebook. Likewise the work-week instantiates its own 7-day epicycle of time, as does the work-day. But here again if you want to use a non-scalar codomain and talk about not only numbers of traffic but geotag it to where it came from, then the work-day epicycles are actually bound to specific spots on the globe and it’s these eigenmoditas that are added up to get the geotagged traffic pattern.

(So on my regular rag about dividing groups only into homogenous equivalence classes, the eigenmodal components have something to add. The Sydney people have a warm-weather mode affecting them when the New York people have a cold-weather mode. But the Christmas eigenmode is affecting them both simultaneously, however not all of the people in either place necessarily celebrate it, yet probably everyone is affected by the tradition.)

• the Mauna Loa CO₂ data. In robust & exploratory data analysis class, Karen Kafadar showed us how to decompose this time series of carbon dioxide measurements at a Hawai’ian volcano into seasonal effects, last-day’s lags, and perhaps a different “mode” that’s picking up on local cause of a greenhouse effect.
• Crime waves. Like, literally, if they’re waves. Perhaps in the summer it’s easier to be outside so whenever the temperature waves up, so does the crime rate. Or perhaps a standing soliton of copycat murders propagates throughout a region. Or in Christian countries, maybe a sufficient number of people in rich neighbourhoods go on vacation that it makes sense to criminals to watch who’s left for the holidays and systematically bust into people’s houses whilst they’re away.
• Some theories of culture posit that "Everything goes in cycles". Not just the Elliott wave theory, but I remember a Mormon friend explaining me his theory (and I’m not sure if this is generally held in Mormon doctrine or not) that the USA was getting too sinful and profligate and irreverent and secular, and that in time, like Sodom and Gomorrah, G-d would lay his country low.
• (I’ve heard other Estadounidenses draw parallels to “the latter days of Rome”, without realising that Rome's fall can't be pinpointed specifically because it didn't Dirac down but rather “crumbled” or slowly changed, which is not their meaning.)
• via @BPS_ON_RISK: Waking up. Definitely an eigenmode, although for some of us the mode might be multiple frequencies (wake @ 7am on Wednesdays, wake @ 11am on Thursdays) or it may shift in different times of life or throughout the week/month (“I was in Manila last week and now back in Mountain View”)

Real numbers are imaginary, and imaginary numbers are real.

[I]maginary numbers describe a physical state of something, so as much as a number can exist, these do. But … real numbers, [being ideal], are imaginary.

David Manheim

(I changed some parts that I don’t agree with but the phrasing and initiative are his.)

The “rational” numbers are ratios and the “counting” numbers are, um, what you get when you count. But “real” and “imaginary” numbers have nothing to do with reality or imagination (each is both real and ideal in the same sense).



How about we start referring to them this way?

• ℝ = the complete numbers. ℝ is the Cauchy-completion of the integers, meaning that ℝ has completely fills in enough options so that any sequential pattern will be able to dance wherever it wants and never need to step its shoe on another element outside the system in order to fulfill its pattern.
• Any field adjoined to the √−1 becomes "twisting numbers". This derives from the “twisting” feeling one gets when multiplying numbers from ℂ. For example 3exp{i 10°} • 5exp{i 20°} = 15exp{i 30°}, they spiral as they multiply outwards. Keep multiplying numbers off the zero line and they keep twisting. Tristan Needham coined the word “amplitwist” for use in ℂ.
• ℂ = the complete, twisting numbers. Since ℂ=ℝ adjoin √−1.
• "Complete spiral numbers" sounds nice as well.

Just to give a few examples of other acceptable numbers systems:

• the algebraics
• sets
• DAGs
• square matrices … with many kinds of stuff inside
• special matrix families
• certain polynomials (sequences) … taking many kinds of things (not just “regular numbers”) as the inputs
• clock numbers (modulo numbers)
• Archimedean fields and non-Archimedean fields
• functions themselves … and the number of things that functions can represent boggles the mind. Especially when the range can be different than the domain. (Declarative sentences can have a codomain of truth value. Time series have a domain of an interval. Rotations of an object map the object to itself in a space. And more….)
• And many, many more! Imagination is the limiting reagent here.

## Sub i, sub j

I hope I can say this in a way that makes sense.

One kind of mathematical symbology your eyes eventually get used to is the Σum over all individuals” concept:

$\dpi{200} \bg_white \begin{matrix} \displaystyle \sum_{i=0}^N \ x_i \\ \\ \displaystyle \sum_i \ x_i \\ \\ \displaystyle \sum_{\text{interesting set}} x_i \\ \\ \displaystyle \sum_i \ \mathrm{weight}_i \cdot x_i \\ \\ \displaystyle \sum_i \sum_j x_{i,j} \\ \\ \displaystyle {1 \over n} \sqrt{ \sum_i (x_i - \bar{x})^2} \end{matrix}$

Yes, at first it’s painful, but eventually it looks no more confusing than the oddly-bent Phoenician-alphabet letters that make up English text.

I believe there is a generally worthwhile pearl of thought-magic wrapped up in the sub-i, sub-j pattern. I mean, a metaphor that’s good for non-mathematicians to introduce into their mind-swamps.

That pearl is a certain connection between the specific and the general—a way of reaching valid generality, without dropping the specificity.



Before I explain it, let me talk a bit more about the formalism and how it’s used. I’ll introduce one word: functional. A functional ƒ maps from a small, large, convoluted, or simple domain onto a one-dimensional codomain (range).

Examples (and non-examples) of functionals:

• size — you can measure the volume of a convex hull, the length of an N-dimensional vector, the magnitude of a complex number, the girth of a rod, the supremum of a functional, the sum of a sequence, the length of a sequence, the number of books someone has read, the breadth of books someone has read (is that one-dimensional? maybe not), the complicatedness (Vapnik-Chervonenkis dimension) of a functional, the Gini coefficient of a country’s income distribution, the GNP of a country, the personal incomes of the lowest earning 10% of a country, the placement rate of an MBA programme, the mean post-MBA income differential, the circumference of a ball, the volume of a ball, … and many other kinds of size.
• goodness / scorebusiness metrics often rank super-high-dimensional things like the behaviour of a group of team members into a total ordering of desireable through less desireable. When businesses use several different metrics (scores) then that’s not a functional but instead the concatenation of several functionals (into a function).
• utility — for homo economicus, all possible choices are totally, linearly ordered by equivalence classes of isoclines.
• fitness — all evolutionary traits (a huge, huge space) are cross-producted with an evolutionary environment to give a Fitness Within That Environment: a single score.
• angle — if “angle” has meaning (if the space is an inner product space) then angle is a one-dimensional codomain. In the abstract sense of "angle" I could be talking about correlation or … something else that doesn’t seem like a geometrical angle as normally proscribed.
• distance … or difference — Intimately related to size, distance is kind of like “size between two things”. If that makes sense.
• quantum numbers — four quantum numbers define an electron. Each number (n, l, m, spin) maps to a one-dimensional answer from a finite corpus. Some of the corpora are interrelated though, so maybe it’s not really 1-D.
• quantum operators — Actually, some quantum operators are non-examples because they return an element of Hilbert space as the answer. (like the Identity operator). But for example the Energy operator returns a unidimensional value.
• ethics — Do I need more non-examples of functionals? A complete ethical theory might return a totally rankable value for any action+context input. But I think it’s more realistic to expect an ethical theory to return a complicated return-value type since ethics hasn’t been completely figured out.
• regression analysis — You get several β's as return values, each mogrified by a t-value. So: not a one-dimensional return type.
• logic — in the propositional calculus, declarative sentences return a value from {true, false} or from {true, false, n/a, don’t know yet}. You could argue about whether the latter is one-dimensional. But in modal logic you might return a value from the codomain “list of possible worlds in which proposition is true”, which would definitely not be a 1-dimensional return type.
• factor a number — last non-example of a functional. You put in 136 and you get back {1, 2, 4, 8, 17, 34, 68, 136}. Which is 8 numbers rather than 1. (And potentially more: 1239872 has fourteen divisors or seven prime factors, whichever you want to count.)
• median — There’s no simple formula for it, but the potential answers come from a codomain of just-one-number, i.e. one parameter, i.e. one dimension.
• other descriptive statistics — interquartile range, largest member of the set (max), 72nd percentile, trimean, 5%-winsorised mean, … and so on, are 1-dimensional answers.
• integrals — Integrals don’t always evaluate to unidimensional, but they frequently do. “Area under a curve” has a unidimensional answer, even though the curve is infinite-dimensional. In statistics one uses marginalising integrals, which reduce the dimensionality by one. But you also see 's that represent a sequence of ∫∫∫'s reducing to a size-type answer.
• variability — Although wiggles are by no means linear, variance (2nd moment of a distribution) measures a certain kind of wiggliness in a linearly ordered, unidimensional way.
• autocorrelation — Another form of wiggliness, also characterised by just one number.
• Conditional Value-at-Risk — This formula $\large \dpi{150} \bg_white \int_{0\%}^{10\%} \mathrm{something} \cdot d \, \mathrm{something}$ is a so-called “coherent risk measure”. It’s like the expected value of the lowest decile. Also known as expected tail loss. It’s used in financial mathematics and, like most integrals, it maps to one dimension (expected £ loss).
• "the" temperature — Since air is made up of particles, and heat is to do with the motions of those particles, there are really something like 10^23 dynamical orbits that make a room warm or cold (not counting the sun’s rays). “The” temperature is some functional of those—like an average, but exactly what I don’t know.


Functionals can potentially take a bunch of complicated stuff and say one concrete thing about it. For example I could take all the incomes of all the people in Manhattan, apply this functional:

$\large \dpi{200} \bg_white {1 \over N} \ \displaystyle \sum_{\text{Manhattanites}} \ \text{ income}_j$

and get the average income of Manhattan.

Obviously there is a huge amount of individual variation among Manhattan’s residents. However, by applying a functional I can get Just One Answer about which we can share a discussion. Complexity = reduced. Not eliminated, but collapsed.

I could apply other functionals to the population, like

• count the number of trust fund babies (if “trust fund baby” can be defined)
• calculate the fraction of artists (if “artist" can be defined)
• calculate the “upper tail risk” (ETL integral from 90% to 100%, which average would include Nueva York’s several billionaires)

Each answer I am getting, despite the wide variation, is a simple, one-dimensional answer. That’s the point of a functional. You don’t have to forget the profundity or specificity of individual or group variation, but you can collapse all the data onto a single, manageable scale (for a time).



The payoff

The sub-i sub-j pattern allows you to think about something both specifically and in general, at once.

1. Each individual is counted uniquely. The description of each individual (in terms of the parameter/s) is unique.
2. Yet there is a well-defined, actual generalisation to be made as well. (Or multiple generalisations if the codomain is multi-dimensional.) These are valid generalisations. If you combine together many such generalisations (median, 95th percentile, 5th percentile, interquartile range) then you can quickly get a decent description of the whole.

Kind of like how thinking with probability distributions can help you avoid stereotypes: you can understand the distinctions between

• the mean 100m sprint time of all men is faster than the mean 100m sprint time of all women
• the medians are rather close, perhaps identical
• the top 10% of women run faster than the bottom 80% of men
• the variance of male sprint times is greater than the variance of female sprint times
• differences in higher moments, should they exist
• the CVaR's of the distributions are probably equivalent
• conditional distributions (sub-divisions of sprint times) measured of old men; age 30-42 black women; age 35 Caribbean-born women of any race of non-US nationality who live in the state of Alabama
• and so on.

It becomes harder to sustain sexism, racism, and to sustain stereotypes of all sorts. It becomes harder to entertain generalistic, simplistic, model-driven, data-less economic thinking.

• For instance, the unemployment rate is the collapse/sum of ∀ lengths of individual unemployment spells: ∫ (length of unemp) • (# of people w/ that unemp length) = ∫ dx • ƒ(x).

Like the dynamic vapor pressure of a warm liquid in a closed container, where different molecules are pushing around in the gas and alternately returning to the soup. The total pressure looks like a constant, but that doesn’t mean the same molecules are gaseous—nor does it mean the same people are unemployed.

(So, for example, knowing that the unemployment rate is higher doesn’t tell you whether there are a few more long-term unemployed people, a lot more short-term unemployed people, or a mix.)
• You can generalise about a group using different functionals. The average wealth (mean functional) of an African-American Estadounidense is lower than the average wealth of a German-American Estadounidense, but that doesn’t mean there aren’t wealthy AA’s (max functional) or poor GA’s (min functional).
• You don’t have to collapse all the data into just one statistic.

You can also collapse the data into groups, for example collapsing workers into groups based on their industry.

(here the vertical axis = number of Estadounidenses employed in a particular industry — so the collapse is done differently at each time point)

Various facts about Venn Diagrams, calculus, and measure theory constrain the possible logic of these situations. It becomes tempting to start talking about underlying models, variation along a dimension, and “the real causes" of things. Which is fun.

At the same time, it becomes harder to conceive overly simplistic statements like “Kentuckians are poorer than New Yorkers”. Which Kentuckians do you mean? And which New Yorkers? Are you saying the median Kentuckian is poorer than the median New Yorker? Or perhaps that dollar cutoff for the bottom 70% of Kentuckians are poorer than the cutoff to the bottom 50% of New Yorkers? I’m sorry, but there’s too much variation among KY’s and NY’s for the statement to make sense without a more specific functional mapping from the two domains of the people in the states onto a dollar figure.



ADDED: This still isn’t clear enough. A friend read this piece and gave me some helpful feedback. I think maybe what I need to do is explain what the sub-i, sub-j pattern protects against. It protects against making stupid generalisations.

To be clear: in mathematics, a generalisation is good. A general result applies very broadly, and, like the more specific cases, it’s true. Since I talk about both mathematical speech and regular speech here, this might be confusing. But: in mathematics, a generalisation is just as true as the original idea but just applies in more cases. Hence is more likely to apply to real life, more likely to connect to other ideas within mathematics, etc. But as everyone knows, people who “make generalisations” in regular speech are usually getting it wrong.

Here are some stupid generalisations I’ve found on the Web.

• Newt Gingrich: "College students are lazy."
Is that so? I bet that only some college students are lazy.

Maybe you could say something true like “The total number of hours studied divided by total number of students (a functional ℝ⁺^{# students}→ℝ⁺) is lower than it was a generation ago.” That’s true. But look at the quantiles, man! Are there still the same number of studious kids but only more slackers have enrolled? Or do 95% of kids study less? Is it at certain schools? Because I think U Chicago kids are still tearing their hair out and banging their heads against the wall.
• Do heterodox economists straw-man mainstream economics?
I’m sure there are some who do and some who don’t.
• The bad economy is keeping me unemployed.
That’s foul reasoning. A high general unemployment rate says nothing directly about your sector or your personal skills. It’s a spatial average. Anyway, you should look at the length of personal unemployment spells for
• Conservatives say X. Liberals say Y. Libertarians think Z.
Probably not ∀ conservatives say X. Nor ∀ liberals say Y. Nor do ∀ libertarians think Z. Do 70% of liberals say Y? Now that I’m asking you to put numbers to the question, that should make you think about defining who is a liberal and measuring what they say. Not only listening to the other side, but quantifying what they say. Are you so sure that 99% of libertarians think Z now?
• The United States needs to focus on creating high-tech jobs.
Are you actually just talking about opportunities for upper-middle-class people in Travis County, TX and Marin County, CA? Or does your idea really apply to Tuscaloosa, Flint, Plano, Des Moines, Bemidji, Twin Falls, Lawrence, Tempe, Provo, Cleveland, Shreveport, and Jacksonville?
• Green jobs are the future!
For whom?
• Alaskans are enslaved to oil companies.
• Meat eaters, environmentalists, blacks, hipsters, … you can find something negative said about almost any group.
Without quantification or specificity, it will almost always be false. With quantification, one must become aware of the atoms that make up a whole—that the unique atoms may clump into natural subgroups; that variation may derive from other associations—that the true story of a group is always richer and more interesting than the imagined stereotypes and mental shorthand.
• What’s wrong with the teenage mind? WSJ.
a teenage mind?
• French women eat rich food without getting fat. Book.
• French parents are better than American parents. WSJ.
• What is it about twenty-somethings? NY Times.

If you sub-i, sub-j these statements, you can come up with a more accurate and productive sentence that could move disagreeing parties forward in a conversation.

Unwarranted generalisations are like Star Trek: portraying an entire race as being defined by exactly one personality trait (“Klingons are warlike”, “Ferengi’s ony care about money”). That sucks. The sub-i, sub-j way is more like Jack Kerouac’s On the Road: observing and experiencing individuals for who they are. That’s the way.

If you want to make true generalisations—well, you’re totally allowed to use a functional. That means the generalisations you make are valid—limited, not overbearing, not reading too much into things, not railroading individuals who contradict your idea in service of your all-important thesis.

OK, maybe I’ve found it: a good explanation of what I’m trying to say. There are valid ways to generalise about groups and there are invalid ways. Invalid is making sweeping over-generalisations that aren’t true. Sub-i, sub-j generalisations are true to the subject while still moving beyond “Everyone is different”.