LLM Papers that I have been recommended

These are papers that I have been recommended with their links. Who knows if they are as good for the Humanities as some people think?

• Edward Hu et al's LoRA: Low-Rank Adaptation of LLMs (Oct 2021).
• The so-called Chinchilla paper. (i.e. Jordan Hoffmann et al, Training Compute-Optimal Large Language Models, Mar 2022); also available at Papers with Code.
• Harm DeVries, Go Smol or go Home blog-post discussing the chinchilla paper.
• Lewis Tunstall et al (October 2023), describing the Zephyr model, a smaller LM "attuned to user intent"
• Suriya Gunisekar et al's Textbooks Are All you Need (June 2023) which describes the phi-1 system that uses higher quality tokens to train smaller models that still perform well (plus synthetic data generated by ChatGPT 3.5)
• Dohnmatob et al's Model Collapse Demystified (February 2024) that explains the problem when training sets reuse their own outputs---giving rise to some of the strategies that DeVries discusses in the README of the bigcode-dataset project.
• Nilesh Barla's blog post on how to train a custom embedding LLM model given the Zephyr model to help generate training data (April 2024).

And if all of these papers make you want to try stuff, consider RunPod ....

Again

Spending a lot of time today thinking about repetition (Difference und Wiederholung, in the words of Deleuze) as problems of knowledge representation. The impetus came from a book on Caterina of Medici (Leah Redmond Chang's Young Queens), which in turns uses the Chronicle of Le Murate to tell the story of eleven-year old Caterina getting caught up in the battle between Medici-Pope Clement VII and the Florentine Republic.

As so often, Sister Giustina Niccolini was writing many years past the events, in 1598, about a situation that happened in the summer of 1530. The rough of it was that the Republican council of Florence wanted to use Caterina as a pain point in their negotiations against the Pope, and the nuns of Le Murate, who were sheltering Caterina, wanted to prevent that, fearing that she might get hurt.

So we have all the usual elements, we have goals and plans and scripts. The interesting part is the conflicts, which interrupt these scripts (or just attempt to do so). 

Silvestro Aldobrandini, the secretary of the council, goes with his soldiers to pick up Caterina from Le Murate in the Via Ghibellina. At this points, plans made by the council and plans made by the pope and his Spanish supporters have been made and thwarted. 

Though the council had placed Caterina there as a guest in 1527 to hold her hostage, they now needed her back for a new plan--to pressure the pope into withdrawing his troops. The nuns refuse to hand her over by refusing to open the door. 

Aldobrandini returned later that night to blast the door to splinters (with canons?), thereby foiling the nuns' plan of keeping Caterina safe in the nunnery. So the nuns physically block Aldobrandini and his men.

Trying to block the men from penetrating deeper into the convent, the nuns sank to their knees, then threw themselves to the ground, crying, praying, pleading, and arguing. 

--Chang, Leah Redmond. Young Queens: Three Renaissance Women and the Price of Power (p. xii). Farrar, Straus and Giroux. Kindle Edition. 

As a quick aside---notice the nightmare of representing this scene, given the plurality of behaviours and the unclear activities (what did they pray? what did they argue), not to mention the loose quantification.

Aldobrandini continues to pursue his goal of pressuring the pope by replanning, but with the nuns bypassed,  Caterina becomes his next obstacle, neither going willingly or shutting up. 

It is silly to assume that Caterina had no patterns for the scenes that she was making, first arguing to remain at the covenant, and then demanding of the world why she was being submitted to such a violent death (Chang, p. xix). So there are scripts here that elude us and that are prolongued. 

Persuading someone is a complex process that may require multiple iterations, some repetition, some novel points, and definitely a good chunk of passage of time. All of this was happening while Aldobrandini was trying to get her on horseback to someplace where she could be used in her role as pressure point. (Chang makes no mention of how that was supposed to work, perhaps her sources did not know either.)

Chang suggests that Aldobrandini must have been swayed, but it is not clear that this is true. Aldobrandini needed Caterina alive, and after chasing the pope's soldiers away, Caterina would most likely return to the covenant, where she had spent the previous three years as hostage. I personally see no giving in when Aldobrandini tells her that she will not come to harm (a dead hostage is a useless hostage) and that he will return her to the covenant within a month's time (her preferred "storage place", as far as the Republican council of Florence was concerned).

Aldobrandini told Caterina that no harm would come to her. He promised to return her to Le Murate within the month. (Chang,  p. xix)

At this point, the whole thing turns into a morality tale of sorts. The Medici restoration leads to the execution of counselors and the torture of those that are banished (Chang, p. xix), but Caterina (somehow) manages to intercede for Aldobrandini and get his death sentence commuted to banishment (with or without torture, Chang does not tell us). 

This then leads us to some of the meta-historical aspects of this story that Hayden White would have immediately jumped on. Clearly the Le Murate chronicle is not a neutral source. Sister Niccolini emphasizes the connection to the French Queen Caterina to the Medici-funded nunnery (Chang, p. 14), proudly recounts the gifts (Chang, p. xix) received, and the ongoing relationship. In fact, Caterina would petition them to pray for her continuously in 1573 and again in 1583 (Chang, p. 17), as we know from her letters (it is unclear whether Niccolini mentions that as well). By the time Niccolini completed her chronicle, the French Queen had been dead for almost a decade (1589).

With that narrative setup, we immediately have heroes and villains, and the morality tale acquires a structure of perhaps a tragedy for Aldobrandini, or the council? Or perhaps even a comedy from Caterina's point of view? More textual analysis would be necessary to make either of that attribution stick. The nuns clearly wanted to be seen as powerful intercessors, either in their arguing or in their praying. 

We therefore need representations that capture these emplotment aspects, the role attributions. Perhaps one might argue that our question of which nun did what is highly misguided. The nuns (and perhaps even the soldiers) function more like the chorus in a Greek theatrical tradition, they as a group embody a sentiment that the author wishes to express. 

Is weak second-order logic sufficient for Humanities research?

In the process of reading Jouko Väänänen's article on "Second-order and Higher-order Logic" in the Stanford (Internet) Encyclopedia of Philosophy, I came across the question of whether weaker forms of second-order logic are decidable or not.

One interesting paper in this regard is Michael O. Rabin's paper from July of 1969, where he shows the decidability of some second-order theories (using the automata on infinite trees, a reduction of a Turing machine that can only read but not write to the tape).

Perhaps most promising seems to me to be that both linearly ordered sets (Corollary 2.2) and countable well-ordered sets (Corollary 2.3) is decidable. The result that Rabin seems to be especially proud of, that the second-order theory of a unary function in a countable domain is decidable (Theorem 2.4), seems to be unhelpful for the knowledge representation needs of the Humanities.

What seems unclear at this point is whether weak second order logic allows the relating of sentences as well as contextualization. That seems to be the crux of the matter for the Humanities, after all. Väänänen writes:

In weak second-order logic we have no function variables and the relation variables range over finite relations only. The resulting logic is in many ways similar to the extension of first order logic by the generalized quantifier $Q_0$. 

The inability to iterate over the functions seems acceptable, as we can turn all finite function applications into terms anyway.

Perhaps it will require a look at finite model theory to make progress on this question.

Addendum: Notice that Daniel Leivant's Higher Order Logic discussion also emphasises that the key features of weak second-order logic is that the relations variables "range over finite relations" [p44], which is later clarified to mean that the relation variable "R is ranging over finite sets" [p.45] and that there are not function variables [p44]. Among the noteworthy results, Leivant mentions that

• Though the upward Skolem-Löwenheim property does not hold, the downard Skolem-Löwenheim property does [p.45]
• The set of f-valid second order formulas is PI-1/1 [p.45].
• Linear ordering and one function are both decidable [p.45].

Some details on the relation between 2nd Order Logic and Set Theory

For Humanities researchers, who relate sentences to sentences, 2nd Order Logic seems more unavoidable than perhaps to knowledge engineers, who prefer to remain decidable. This tension forces me to revisit the precise ways in which 2nd order and first-order formalisms influence expressivity.

Now I found a nice quote in the Stanfard Encyclopedia Of Philosophy article on Higher-Order Logic about the relationship between 2nd-order logic and set-theory that illuminates some of the puzzlement about that connection:

Perhaps this [result of comparing the model hierarchy and the Levy hierarchy, RCK] would not be so puzzling if we thought of set theory as a very high order logic over the singleton of the empty set. After all, set theory permits endless iterations of the power set operation, while second-order logic permits only one iteration.

Applying Peirce to Image Interpretation --- Part Two

If we look again at our four categories --- NONE, ALL, UNIQUE and SHARED --- we immediately realize that they are all grappling with a disjunction that is the enumeration of our sources. Recall that we said that Altdorfer is facing the following situation (in the single source case):

Rufus v Arrian v Justin v Schedel v Harlieb v Aventinus

(where v is the usual vel of classical logic, inclusive or): a big disjunction of choices.

Thus, if we see 32k as the number of battle participants on the flag of the Macedonian infantry, we know that it is either Justin or Schedel or Aventinus, but not Rufus, Arrian or Hartlieb.

The problem is that this representation is logically incomplete. What is missing is the precondition that these are in fact all of the possible sources on the Issos battle that Altdorfer could have consulted. So, the more correct logical form would be something akin to (now in sentential representation):

(and  (sourceUsedInCW Altdorfer-Alexanderschlacht-1528 ?X)
  (memberOf ?X (TheSet Rufus Arrian ... Aventinus))
  (not 
     (thereExists ?Y
        (sourceUsedInCW Altdorfer-Alexanderschlacht-1528 ?Y)
        (not (memberOf ?Y (TheSet Rufus Arrian ... Aventinus))))))

Since this construction is so awkward, we usually have syntactic sugar to make this more palatable.

(candidateSourceUsedInCW Altdorfer-Alexanderschlacht-1528 Rufus)

...

(candidateSourceUsedInCW Altdorfer-Alexanderschlacht-1528 Aventinus)

And then some meta-predicate that makes the same point

(completeExtentAsserted candidateSourceUsedInCW)

... most likely in a suitably restricted microtheory context.

While this is inferential successful and will allow us to conclude NONE in the case of the 12k killed Persian foot, or UNIQUE in the case Oxarthes and the Frauenzimmer, it is also historically false. We have in fact no way to know that there are no other historical sources that were available to Altdorfer at that time. In fact, any day a chance find in the Regensburg library, comparable to the one that recently uncovered the Quintus Rufus edition that Aventinus apparently used (cf Wagner-Jehle, Albrecht Altdorfer: Kunst als Zweite Natur, in: Regensburger Studien zur Kunstgeschichte, Bd 17, Regensburg 2012), might reveal another source.

This could directly affect our UNIQUE or NONE queries, or turn an ALL into a SHARED. It might also upend our source dependency analysis in the case of a former SHARED query. 

This is where truth maintenance comes into play. We must at all times be prepared to take this enumerated set extents and re-validate them against the latest state if historical research, and modify our claims accordingly. The specific transformations are as follows (if there is any change, rather than no change):

• NONE into UNIQUE if the new source contains the feature
• UNIQUE into SHARED if the new source contains the feature
• ALL into SHARED if the new source does not contain the feature

Only SHARED would continue as SHARED, albeit potentially influencing the dependency graph. For example, in a situation before Cord Meckseper's Iconography Paper, discussed in Part One, where only Justin's Epitome and Aventinus' Bavarian Chronicle might be in consideration as source for the 32k Macedonian foot, one might be tempted to think that Aventinus used Justin; but the insertion of Schedel's World Chronicle makes opens up new possibilities here.

Applying Peirce to Image Interpretation --- Part One

The basic notion of Charles Sanders Peirce's, that the positing of an abduced structure of some sort (e.g. a Schankian script in a Davidsonian representation world) can organize a set of disparate pieces into an explanation, is sound. The problem is how to play that through for the specific case of, say, Albrecht Altdorfer's Alexanderschlacht.

The start of the story must be the flags and the labels, because they give specifics. We also need to hold the inscription aside, because there is a strong suspicion that it is not by Altdorfer himself, but later (possibly by Jean De Pay in 1658, whose restauration receipt is already mentioned by Franz von Reber's Catalogue of the Old Pinakothek's collection of 1900 in the 8th edition, though Buchner in the 1938 Katalog for the 400th anniversary exhibition in Munich seems first to suggest that this included a reworking of that inscription).

There is a list of possible sources that Altdorfer could have used for obtaining the details of the battle. That list includes Arrianus, Quintus Curtius Rufus, Justin's Epitome of Trogus, Aventinus, the Alexander Romance of Johannes Hartlieb, Schedel's Nuremberg Chronicle, and Plutarch. (For some of Aventinus' views on this matter, see this post here.)

Per CSP's recommendation, we can now construct alternate worlds of various forms. The basic idea would be that Altdorfer used exactly one source. For example, as Cord Meckseper did in his article on the iconography of the battle (Zeitschrift des Deutschen Vereins für Kunstwissenschaft, XXIII, 3/4, 1968, 178-185), one might assume that Schedel's Weltchronik might suffice.

The point is to expand what »suffice« might mean in terms of a strategy of mechanical interpretation. We assume that the narrative (it is not very long in Schedel) is summarized in a suitable Davidsonian event representation. This instantiation then identifies actors, actions, matériel and similar.
We also need a similar instantiation of the content of the painting itself, as an event, describing the various discernibles. Here, we can simplify the problem by delineating (e.g. with TEI standard bounding boxes) and describing at the action level (moving, turning, etc.) all those entities that any of the art historic works we have considered have identified, i.e. the superset of the interpretation constituents. Recall that we wish to model the argumentation, not automate the argumentation.

But if we have the description of the picture, and the description of the events as reported in the various sources, what then is the abduction doing for us? It is the mapping between these components, a representational correspondence. The picture has a golden-armored man riding a white horse. The story has Alexander leading a charge (say, not all sources go into that detail). The abduced mapping then says that the man on the white horse is Alexander (and by one step of inference that leads us beyond the sources, the horse is Bukephalos).

Consider the battle flag for a moment, which consist of the following pieces of information:

• the army to which the troops are attached (Greek/Macedonian or Persian, often mentioned pars pro toto via the commanders Alexander and Darius III)
• the type of troops we are talking about (here, mostly foot vs horse, i.e. infantry vs cavalry)
• the number of men present at the beginning
• the cardinalities of the categories at the end of the battle
• killed (presumably this includes missing in action, in modern parlance)
• captured (e.g. 40k Persians in Justin and Schedel)
• escaped (1k horse w/ Darius in Quintus Curtius Rufus, 4k in Arrian)

Here we are matching not against actors or roles in the script, we are matching against event statistics, so to speak, or statistics of sub-events (e.g. the troop numbers at the beginning and at the end of the battle). The flags visually represent the battle statistics (which are abstractions and therefore not visible per se).

In all of these representational mappings, we can run into four situations:

• NONE: there is something in the image that is in none of the sources (the scythes on the fleeing chariot, the chariot driver felled by the arrow, the boats on the sea), 
• ALL: there is something in the image that is in all of the sources (Alexander, Darius)
• UNIQUE: there is something in the image that is only one of the sources (the women, Oxarthes)
• SHARED: there is something that is in some, but not all of sources (the 32,000 Macedonian foot in Aventinus, Justin's Epitome of Trogus and Schedel's Weltchronik)

We have different reactions to these situations, then, as they give us different information about the question of which sources were used.

• NONE means that Altdorfer invented the matter himself
• ALL means that we cannot distinguish between the different source theories based on this piece of information
• UNIQUE means that the specific source was definitely used

SHARED has a special status, as it is often an indication of the genealogy of the sources themselves. For example, Altdorfer was not the only one working from sources, both Aventinus and Schedel had to use ancient reports to underpin their reconstructions. We may not be able to separate out Justin from Trogus, since Trogus did not survive the ravages of time as an independent source. But since Justin  (200-400 AD) is temporally before Schedel (1493) and Aventinus (1526), it is likely that Schedel used Justin (in the particular case of the 32k Macedonian foot). There are other cases were Schedel agrees with Justin, such as the number of 40k captured Persians.

The problem is trickier in the case of Aventinus, whose Bavarian Chronicle was published in 1526 when Aventinus was already living in Regensburg. Since Schedel predates Aventinus by over 30 years, and Aventinus only discusses Quintus Curtius Rufus as well as Arrian (Bk I, Cpt 153, p.337), as well as disparaging Hartliebs Romance. So perhaps Aventinus used Schedel and not Justin; unfortunately he mentions neither.

One more point about the NONE category, because Alexander's horse Bukephalos falls into it. None of the source I have looked at specifically mention that Alexander was riding Bukephalos at the battle of Issos. But Bukephalos is not invented in the same way that the chariot driver pierced by the arrow is, and not in the same way that the scythes on the carriage are. So we will have to return to the problem of this category at a later point.

But before that, we have to look at the problem of dealing with the open-ended disjunctions.

Rule Probabilities and Quantification

In Artificial Intelligence of the symbolic kind, it is often the case that probabilities are applicability constraints that are simply attached to the rules.

[In some survey of some part of the populace,] 80% of all Biritsh scientists are the children of scientists [the oldest child, an only child, etc].

The problem with these types of examples is that they disguise how the probabilities end up affecting the quantifier structure. This is especially true when we deal with a language that supports exceptions (i.e. non-monotonic changes) in the knowledge capture because there the extremes are best captured as exceptions.

None of the cities of of the Roman empire, excepting Rome itself, had more than five hundred thousand citizens.

(exceptWhen (equalSymbols ?CITY Rome)
    (=> (and (isa ?CITY City) (subOrganization ?CITY RomanEmpire)
             (population ?CITY ?POP))
        (lessThan ?POP 500000)))

(with appropriate temporal restrictions somewhere, e.g. implied by the Roman empire membership or on the  population number).

At the same time, at the middle range of the distribution, it would be most natural to speak of existential quantification, perhaps baths or amphitheaters, or a temple dedicated to a specific deified hero or an oriental fertility cult.

Thus we observe that a continuous model (probabilities [0,1]) meet a discrete model of quantifiers and exceptions, and we expect the continuous model to deal with data changes more gracefully, especially if the representation of discrete parts is not automatically driven by continuous representation. Perhaps we could consider 1% and 99% as the cutoffs for 'none' and 'all' (not in principle, in a specific case), and have rules to then conclude the discrete quantifier structure from the data distribution and its interpretation?

Improving Scripts

For some scripts, it would seem helpful to know not merely which scenes follow which scenes, but also what the temporal constraints are on the individual scenes to get them accomplished.

Consider Arthur C Danto's example of lighting a pipe; if the match burns at the standard rate, then there are just a few moments until the match is too hot to hold and needs to be dropped. Similarly, for a recipe, there are time constraints on how soon the next processing steps have to take place before the processed food has changed in temperature or composition or moisture contents to make the next step less successful (or even impossible).

At the same time, similar constraints hold in the other direction of moving "too early" as it were. The rates of these changes put temporal bounds on the script that would be good to know and good to have.

Ideal Types continued

While it seems perfectly normal to have Restaurant as an ideal type, it seems less convincing to have a restaurant visit as an ideal type; yet this is what the script was.

So we first note that there is a difference between ideal types and scripts in the sense that ideal types are much more structural. Ideal types are concerned about the fact that restaurants have kitchens and waiting areas and menus and specials of the day and a parking lot and fire exits. For the scripts, the emphasis is on the event types that can take place in that setting. The focus is less on how to build a restaurant and more on how to behave within one.

The second aspect is that ideal types are themselves structurally developmental. Sometime in the modern era, probably increasingly so after WW2, restaurants acquired cold storage rooms to house the meats and other perishables between acquisition and preparation. The ideal type can accommodate that shift readily; there are temporal constraints on the presence of the cold room until pretty much all restaurants have them (and use them to great effect, e.g. Ratatouille).

It is not clear that scripts as encodings of expected behavior accommodate as quickly. Perhaps temporally recent changes that affect the behavior more are required to show script changes; for example, the elimination of smoking rooms or requests to silence cell phones during dinner.

Perhaps there is then a general tendency as to classify the ideal type as part and parcel of structural history (Strukturgeschichte), the background on which the events take place (some of which may very well be guided by scripts).