It also does an amazing job of taking snapshots of posts. I use it all the time to grab posts from HackerNews so I can mark them up.
And it automagically syncs everywhere across devices and lets me store way too many files on the web like the ADD packrat I am, without breaking a sweat.
In short, this software just works, and it works well.
So when somebody behind Zotero talks about how to develop software, I listen.
And it's a fun post with some history. You should save this post to Zotero, and then read it.
> But that slow formation led to software that was durable rather than ephemeral, with a strong foundation that could be built upon.
People say that agentic development is great because you can churn out so much so fast. But that doesn't mean that any of it will be truly good and reliable.
The things that are truly insightful and solid end up being used exponentially more, which makes the linear cost of extra development time (asymptotically) insignificant in the cost/benefit equation.
The emphasis on quantity while neglecting quality calls to mind a passage from EWD1175[0]:
> My second warning remark is that I shall refuse to discuss the academic enterprise in financial terms. The first reason is that the habit of trying to understand, explain, or justify in financial terms is unhealthy: it creates the ethics of the best-seller society in which saleability is confused with quality. The other day we had to discuss the professional quality of one of our colleagues, in whose favour it was then mentioned that one of his Ph.D.s had earned lots and lots of money in the computer business, and few people seemed to notice how ridiculous a recommendation this was. We also know that the financial success of a product can be totally independent of its quality (as everyone who remembers for instance the commercially successful IBM360 should know). The second reason for my refusal is that the value of money is a very fuzzy notion, so fuzzy in fact, that efforts to understand in financial terms always lead to greater confusion. [Remember this, for it is quite likely that this afternoon will give you the opportunity to observe the phenomenon. Note that money need not be mentioned explicitly for the nonsense to emerge, a reference to "the taxpayer" can do the job. The role of "the taxpayer" then invariably leads to the conclusion that of State Universities at least the undergraduate curriculum has to be second- or third-rate.] The final reason for my refusal is that the habit appeals to the quantitative mind and I come from a culture in which the primarily quantitative mind does not evoke admiration. [A major reason that we considered Roman Catholics to belong to a lower class was precisely their quantitative bent: they always counted, number of faithful, number of days in purgatory, you name it.....]
It may be so but the article does not claim that the strong foundation is the code, rather it seems to be product design, and design of other products, at that (the two predecessors, webapp and desktop app). No reason why you couldn't study existing products now and tell your agent to build something based on that.
This feels like a sleight of hand to me. The hard part of evolving Scribe and Web Scrapbook was discovering that a browser extension manipulating a local SQLite database was _the only_ architecture that could reconcile local offline persistence with live DOM scraping across arbitrary catalogs of academic data.
An agent can synthesize existing solutions but (because I see this failure mode at work constantly) it can't synthesize an architecture to resolve the sorts of tensions that the person prompting it doesn't yet understand (not that that is stopping anyone). You can't prompt it to build something if the operational primitives required to solve the problem haven't been mapped.
"Build a tool based on Scribe and Web Scrapbook" in 2003 would've made a fragile PHP wrapper because that's what the existing landscape looked like.
Yes, exactly. And this is why I don't think that the LLMs can make significant process beyond what humans have done and published.
"But the math proofs," people will say. A lot of those seem to be spam-solving things with a huge swath of existing lemmas, and a some of these are being debunked and retracted.
Just today I was quizzing ChatGPT about a basic grammar question for a language that has huge training data but for which the grammar was not well documented. It kept giving me confidently wrong answers until I drilled and drilled it and then finally it found/gave back an explanation that perfectly fit a pattern given in one particular grammar, citing that as a source. It doesn't appear to have been able to figure out the inner structure on it's own. It appears only able to pattern match and put things together from what humans have already discovered and written.
Right, there's the difference between statistical interpolation and semantic induction. The whole point is that LLMs can't reason from first principles to drive missing rules. It keeps confidently feeding you approximations until it collides with some source that already mapped it.
I love great software and agree that great software is evolved - not built. The whole point of building v0.1 is to figure out what's wrong and should be fixed in v0.2
At the same time, there are broadly three motions in the loop:
- thinking/discussing (what should it do)
- building (Make it do that)
- using (Seeing whether that is actually what it SHOULD do)
And then of course you repeat until you run out of time, money, patience, volition, etc. For some software there is a terminal state - it truly does exactly what it should. For most you're always reaching for it.
LLMs definitely accelerate 2 and potentially can help accelerate 1 and 3. As such cycle time can be reduced. It still may take 100 turns to get what you want, but I'd be surprised if the clock time for the 100 turns would be unchanged using AI.
good things take time because they grow from something like seed. as that seed grows, it figures out its local and global context. a curious and patient caretaker of this seed will spend a lot of time looking at it, understanding it, trying to figure out the right way to give the small plant a steady foundation. with care and attention, it could grow into a tree and attract all sorts of other insects, animals, and all sorts of life.
speed kills quality. it’s literally impossible to make anything good fast.
we know this, and it still applies to software. while we may be able to make things faster, they will never become good (or great) without an incredible amount of care, patience, and joy from its maker.
there are no shortcuts to quality. it will always take a lot of time to make anything good.
I am not sure it IS impossible to make anything good fast. Look at how many incredible songs have been made in a single afternoon, or fantastic ideas for a simple product came to somebody in an instant.
Your comment made me think two things:
• Sometimes constraints make things better, and ‘speed’ can occasionally make you prioritise the things that actually matter so you deliver stuff that counts
• Sometimes, thinking longer about something doesn’t get you closer to the correct answer. You can rearrange and refactor and rewrite and redesign, but you won’t always get something objectively better than what you originally came up with. It’s still your thoughts, your brain and your ways of working that shape the output (and they haven’t changed).
Investing more time to make something better should be a conscious decision. Perhaps it’s one people decide against for the wrong reasons.
But sometimes things need something other than time and effort spent to make them better.
The reason great artists spend so much time developing their skills is that the result no one knew they wanted comes from skill and the new possibilities emerging from that.
So as a software developer now might be the best time to throw system design on it's head and develop without a direction but make sure everything is as good as one can forge it.
Surprising functionality and stuff not found elsewhere then more or less simply emerges from that.
Granted - that has a certain freedom and no pressure to make money as a prerequisite but so do the 5 years noted here.
As somebody responsible for the acquisition of users and growth of a company in terms of customer and revenue, this is a VERY salient point:
> “… we could not have accelerated Zotero’s conception, because we did not know exactly what we wanted, and so could not have written coherent prompts for an LLM.”
A surprising proportion of software products, maybe even businesses today, are solutions in search of a problem.
Sometimes that’s okay, but only sometimes. And being a solution in search of a problem requires you to get everything /else/ pretty much perfect if you want to succeed.
The fact Zotero paid attention to what people wanted, and gave it to them, and were market oriented, is demonstrably a big part of their success.
It is MUCH easier to make something people want, than to make them want something you made.
Agreed, but it is also much easier to get something made in the first place. LLMs enable rapid prototyping and rapid shifting to solve real problems. My applications are morphing from mediocre to highly useful problem solving machines much more rapidly now.
That’s a really good point! But I am always surprised by how often people will avoid putting their prototypes out there and let the market shape their product.
Rapid prototyping is an amazing opportunity afforded to us by AI. But some people use that potential to spend even longer on a more developed prototype that they are too attached to to get feedback on!
> If AI had existed in the early aughts, we could not have accelerated Zotero’s conception, because we did not know exactly what we wanted, and so could not have written coherent prompts for an LLM. Instead, it took a great deal of time and collaboration to develop a clear vision for what Zotero should be.
AI doesn't preclude this. In fact, it can help accelerate parts of it.
He's describing the typical big project lifecycle:
- Examine the landscape
- User research (how they use existing software, what their frustrations are, etc)
- Brainstorming
- Early ideas and prototypes
- Refinement, user feedback
- Solidify the vision and high level process design
- Choose technologies
- Design & architecture
- Plan out phases
- Build phases, then test them with users
LLMs are great at research, and great at prototypes. Once you have your design, they're good at coding as well. They're also good at distilling user feedback.
A very very strong point. I have personally detailed entire projects because adding a feature seemed easy with AI. It is very difficult to vibe code and not add a bunch of useless crap features.
Really? I find LLMs quite bad at deleting code. If you ask them to add a feature, then later take it out again, the codebase still grows. It always grows. Every time I've tried it, llms have failed to simplify code via refactoring. Even fable is incredibly bad at this for some reason.
LLMs are excellent at making prototypes though. And prototypes can be an excellent way to stop yourself from implementing the wrong features in the first place.
I guess we'll only be able to verify this 10-15 years after coding agents arrived. Personally, I think there will always be a market for apps created with care and great attention to detail.
This post is about Zotero. https://www.zotero.org/
If you are not an academic, you might not know Zotero.
It is such a pleasure to use. Every app should be like this.
I read everything in it, including books I'm going through right now from https://teachyourselfcs.com/
It also does an amazing job of taking snapshots of posts. I use it all the time to grab posts from HackerNews so I can mark them up.
And it automagically syncs everywhere across devices and lets me store way too many files on the web like the ADD packrat I am, without breaking a sweat.
In short, this software just works, and it works well. So when somebody behind Zotero talks about how to develop software, I listen.
And it's a fun post with some history. You should save this post to Zotero, and then read it.
People say that agentic development is great because you can churn out so much so fast. But that doesn't mean that any of it will be truly good and reliable.
The things that are truly insightful and solid end up being used exponentially more, which makes the linear cost of extra development time (asymptotically) insignificant in the cost/benefit equation.
> My second warning remark is that I shall refuse to discuss the academic enterprise in financial terms. The first reason is that the habit of trying to understand, explain, or justify in financial terms is unhealthy: it creates the ethics of the best-seller society in which saleability is confused with quality. The other day we had to discuss the professional quality of one of our colleagues, in whose favour it was then mentioned that one of his Ph.D.s had earned lots and lots of money in the computer business, and few people seemed to notice how ridiculous a recommendation this was. We also know that the financial success of a product can be totally independent of its quality (as everyone who remembers for instance the commercially successful IBM360 should know). The second reason for my refusal is that the value of money is a very fuzzy notion, so fuzzy in fact, that efforts to understand in financial terms always lead to greater confusion. [Remember this, for it is quite likely that this afternoon will give you the opportunity to observe the phenomenon. Note that money need not be mentioned explicitly for the nonsense to emerge, a reference to "the taxpayer" can do the job. The role of "the taxpayer" then invariably leads to the conclusion that of State Universities at least the undergraduate curriculum has to be second- or third-rate.] The final reason for my refusal is that the habit appeals to the quantitative mind and I come from a culture in which the primarily quantitative mind does not evoke admiration. [A major reason that we considered Roman Catholics to belong to a lower class was precisely their quantitative bent: they always counted, number of faithful, number of days in purgatory, you name it.....]
[0] https://www.cs.utexas.edu/~EWD/transcriptions/EWD11xx/EWD117...
An agent can synthesize existing solutions but (because I see this failure mode at work constantly) it can't synthesize an architecture to resolve the sorts of tensions that the person prompting it doesn't yet understand (not that that is stopping anyone). You can't prompt it to build something if the operational primitives required to solve the problem haven't been mapped.
"Build a tool based on Scribe and Web Scrapbook" in 2003 would've made a fragile PHP wrapper because that's what the existing landscape looked like.
"But the math proofs," people will say. A lot of those seem to be spam-solving things with a huge swath of existing lemmas, and a some of these are being debunked and retracted.
Just today I was quizzing ChatGPT about a basic grammar question for a language that has huge training data but for which the grammar was not well documented. It kept giving me confidently wrong answers until I drilled and drilled it and then finally it found/gave back an explanation that perfectly fit a pattern given in one particular grammar, citing that as a source. It doesn't appear to have been able to figure out the inner structure on it's own. It appears only able to pattern match and put things together from what humans have already discovered and written.
At the same time, there are broadly three motions in the loop: - thinking/discussing (what should it do) - building (Make it do that) - using (Seeing whether that is actually what it SHOULD do)
And then of course you repeat until you run out of time, money, patience, volition, etc. For some software there is a terminal state - it truly does exactly what it should. For most you're always reaching for it.
LLMs definitely accelerate 2 and potentially can help accelerate 1 and 3. As such cycle time can be reduced. It still may take 100 turns to get what you want, but I'd be surprised if the clock time for the 100 turns would be unchanged using AI.
speed kills quality. it’s literally impossible to make anything good fast.
we know this, and it still applies to software. while we may be able to make things faster, they will never become good (or great) without an incredible amount of care, patience, and joy from its maker.
there are no shortcuts to quality. it will always take a lot of time to make anything good.
Your comment made me think two things:
• Sometimes constraints make things better, and ‘speed’ can occasionally make you prioritise the things that actually matter so you deliver stuff that counts
• Sometimes, thinking longer about something doesn’t get you closer to the correct answer. You can rearrange and refactor and rewrite and redesign, but you won’t always get something objectively better than what you originally came up with. It’s still your thoughts, your brain and your ways of working that shape the output (and they haven’t changed).
Investing more time to make something better should be a conscious decision. Perhaps it’s one people decide against for the wrong reasons.
But sometimes things need something other than time and effort spent to make them better.
So as a software developer now might be the best time to throw system design on it's head and develop without a direction but make sure everything is as good as one can forge it.
Surprising functionality and stuff not found elsewhere then more or less simply emerges from that.
Granted - that has a certain freedom and no pressure to make money as a prerequisite but so do the 5 years noted here.
> “… we could not have accelerated Zotero’s conception, because we did not know exactly what we wanted, and so could not have written coherent prompts for an LLM.”
A surprising proportion of software products, maybe even businesses today, are solutions in search of a problem.
Sometimes that’s okay, but only sometimes. And being a solution in search of a problem requires you to get everything /else/ pretty much perfect if you want to succeed.
The fact Zotero paid attention to what people wanted, and gave it to them, and were market oriented, is demonstrably a big part of their success.
It is MUCH easier to make something people want, than to make them want something you made.
Rapid prototyping is an amazing opportunity afforded to us by AI. But some people use that potential to spend even longer on a more developed prototype that they are too attached to to get feedback on!
AI doesn't preclude this. In fact, it can help accelerate parts of it.
He's describing the typical big project lifecycle:
- Examine the landscape
- User research (how they use existing software, what their frustrations are, etc)
- Brainstorming
- Early ideas and prototypes
- Refinement, user feedback
- Solidify the vision and high level process design
- Choose technologies
- Design & architecture
- Plan out phases
- Build phases, then test them with users
LLMs are great at research, and great at prototypes. Once you have your design, they're good at coding as well. They're also good at distilling user feedback.
It can also slow it down as people get distracted experimenting with features they can build quickly.
LLMs are excellent at making prototypes though. And prototypes can be an excellent way to stop yourself from implementing the wrong features in the first place.
Well, yes. I mean, it's in the name Generative Pre-trained Transformer: they're text generators!
To delete text using a text generator, you have to emit the original thing taking care to omit the deleted stuff during emission. It's more work.
https://assets.buttondown.email/images/93382906-4996-445c-81...
is just of some passionate academics working on a project with no real economic or social media incentives driving it.