parliament32
Born on June 23, 2013•4649 Karma
Rather, it did work at milestone 14, but then regressed at milestone 15, where it changed the link from a wikimedia image to a nonexistent file in /assets (despite still having the "Photo via Wikimedia Commons" caption).
edit: they removed it :^)
My question, though, is why the "Live, public build log" only showing up to milestone 3, but the artifacts go up to milestone 15? And there are different index.html pages in the artifacts list, one for milestone 14 and one for milestone 15? Are there different conceptions of "milestone" in here? What's up with that?
There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something completely else) and "predict" the next token, and repeat for as long as you want, with the volatility you ask for. There is no control plane, and there never will be a control plane, because asking for that is akin to asking "how do I separate data from instructions when I speak to a person?". You can ask nicely, "pretty please obey the first part of what I say and not stuff after", but there's no way to guarantee it (like you're used to with software). There is just input and output.
The way llms are right now, and the way humans are, there is no side channel.
It's all about training, but even with extensive training, output breaks down if it's probability based and not hard logic and state machine.
It can be done, but unsurprisingly it looks exactly like microservices distributed auth (also ZTP).
It's all the same problem, just instead of a JVM, it's an LLM.
Like in the banking world, you can make everything super authenticated, but if you have an API that receives the latest wire transfer YOU received with the message attached, you don't control the message content and it can be an attack vector.
Being authenticated/authorized is not the solution, it is data that the user can access.
But that doesn't mean that separation between instructions and data is impossible. You can format them in different ways, and you can prevent the output tokens from ever using instruction formatting.
Agreed.
> But that doesn't mean that separation between instructions and data is impossible.
Yes it does! The comments you are replying to are concerned that it is not possible to be sure that data and instructions have been separated. With certain kinds of automated systems (traditional ones), unless you write them incorrectly, you can be sure of this. And it is possible to engage in a productive incremental process where mistakes can be identified and removed, in a way people comprehend and can plan around.
LLMs do not have this. They have heuristics and guesses. Nobody knows what will work ahead of time, nor even a probability that it will work. That is not a doomer comment by the way! The same is true when you talk to a person. But it is a fundamental limitation, it cannot be removed.
Can you make sure the instructions and data are separated and the machine follows only the instructions and doesn't change its behavior based on the data? No.
But the part that's impossible is not "the instructions and data are separated". The part that's impossible is "the machine follows only the instructions".
Separating instructions and data is not impossible, but it doesn't solve your problems.
One really important consequence of this is that even if the data doesn't have anything that looks like instructions, it can poison the machine anyway! If you get too focused on "instructions" then you miss that security flaw!
Even if you don't give the machine any data at all, it might not follow the instructions. It's not instruction/data conflation as the root cause, it's that instructions don't really work in the first place.
Even if special tokens are used absolutely perfectly (somehow avoiding escapes or ambiguities or reflected attacks) they are ultimately the same as highlighting all the parts of the document in different colors. You've saved the signal, but there's no mind to receive the intended meaning.
This means that your markers--while far more exclusive--ultimately exist on the same data-level as punctuation and using ? to indicate a question.
> you can prevent the output tokens from ever using instruction formatting
The right words may still outweigh the formatting around them, the same way that they can already outweigh other words around them.
Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue.
All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!
Likewise, model-actions (e.g. open file) could be written only in red, and hence you'd never learn to do them from the unfiltered data.
The only connection between the red world and the blue world would be the processed trainign chats containing red and blue data togethers -- allowing the model to learn the relationship between them (while only being exposed to examples where red instructions are strictly followed, whatever the blue says)
I don't think is guaranteed to actually work, it's a hypothetical after all, but maybe it's better than the current setup of pushing instructions and data into the same slot.
Think of how an image of a car and a car in front of you may look indistinguishable in 2D -- but due to your 3D vision you know they're not the same thing (but also know the image is of a car, while not literally being a car).
Likewise, blue tokens are the image of red tokens.
You're saying that a Harvard architecture computer can't exist because instructions and data are stored in the same memory, well guess what, in Harvard architecture computers they're not.
Meanwhile.. have you ever paid for a vibe-coded anything? Why would you, when you (along with everyone else) can slop the same thing together in a weekend with a $20 CC subscription?
HN really needs a containment board.
Your verbosity and sentence structure are not a problem. I hope that publishing this gives you a bit more confidence in your writing, because it's legitimately good.
Incentives are entirely different. And really now I am starting to think that Nasdaq maybe should not have index it runs in the first place...
This whole story is about Nasdaq (company) specifically dangling inclusion into the Nasdaq-100 (index) as a means to get SpaceX to list on the Nasdaq (market). They're uniquely able to do this by owning a market and also an index that people care about.
NYSE couldn’t really do this because its own indices don’t matter much. FTSE Russell could theoretically make FTSE 100 inclusion easier to help attract a company to list on the London Stock Exchange, but SpaceX choosing London as its primary market would be odd. S&P Dow Jones Indices has no equivalent incentive, because it doesn’t own a listing venue; its main asset is the credibility of the S&P 500.
In all, this entire story has been about Nasdaq specifically being willing to weaken their index rules in order to attract SpaceX to their market.
As someone who has little experience in American stocks and index, would you explain it a bit more ? What you mean that nasdaq is also the market ?
I thought they are both index, a valuation of entreprises and that's all.
So Nasdaq owns the company which facilitates this trading of stocks. But they also own the company which says what are 100 most important companies on that market.
Now they changed rules to get big new most likely popular stock on their market. This could at least maybe get some new brokers in. Or make them in general more desirable market to be connected to and thus get fees.
I am not just sure if there is even more fees in some part I don't know there...
That's a serious conflict of interest.
This, I think, is the part that irks me the most. Companies adding token-usage-KPIs for engineering is one thing, but when they have to resort to deliberately tricking users into using their slop-generators.. something has gone very wrong, and they're trying very, very hard to make it seem like it's not so.
My personal pet peeve is Copilot in Teams. Did you know, if you turn off Copilot in Teams at an org level, it disables meeting recording entirely? Ignoring that meeting recording has been a core feature dating way back before Copilot-anything, I can't fantom any possible reason why recording a video of a meeting would require an LLM. Transcription, maybe I could see, but that feature is easily togglable with or without Copilot. But if you want to record a meeting, for whatever reason, you need to have Copilot on.
Shenanigans like this is why user counts for LLM features should always be taken with a grain of salt.
Step 2: Complain about how the OSS/Chinese/whatever models are doing releases without approval
Step 3: Prohibit, because "safety" and "financial risks"(?)
So this is the door-shutting Altman et al have been pushing for eh?
https://deepmind.google/models/gemma/gemma-4/
https://developer.nvidia.com/ai-models#:~:text=NVIDIA%20Nemo...
https://www.microsoft.com/en-us/research/blog/phi-4-reasonin...
I'm 99% sure it was one-and-done, box ticked, and now they can be mentioned in comments like this.
Should they be interested in advancing state of the art open models?
Generally, it is conspicuous how American companies are absent when it comes to state of the art open models. Meta tried for some time but it seems they've given up.
For AI, the most profitable part of the value chain is selling inference. None of the big American companies want to release a leading edge model as open source because this would drive the price of inference to $0. Meanwhile, open source AI models are a huge strategic initiative for China. Having commodity Chinese models that are as good as the leading edge American models from 6 months ago forces the American companies to keep paying more and more money to train better and better models since the amount of time they can collect rent on a model they've previously trained is limited to 6 months.
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
Meta/Llama: "What am I, chopped liver?"
I thought the thing keeping inference above $0 was the hardware, and even if that were free there's still the tyranny of the Landauer Limit.
The notable exception is of course the google play services, which is also strategic (they control the OEMs with this, among other things).
And the drivers, but that's mostly not them I think (they could possibly have required open source drivers though)
"The world doesn't go round. It flips over!"
scraping CoT won't stop the advance of Chinese models. neither will a US "ban" on using such models. at this point I'm cheering for DeepSeek or Qwen to catch up to Anthropic. I support anyone who releases open weights.
I strongly recommend open-weight wherever you can. assume any data you pass to a closed model (including opinions or political positions you intimate) will be retained and analyzed in unfriendly ways, either now or ten years from now.
having open-weight models allows users to use/modify them in novel ways.
I don't trust Dario Amodei, Sam Altman and Elon Musk to act in my best interests. Closed models will have an incredible centralizing effect, and concentrate power like we've never seen since the feudal ages.
If you want to see what it's like for the economy to collapse into a single, extremely valuable commodity, under the control of a small elite, look at Saudi Arabia.
also, I just value freedom tremendously. I want to tinker with model weights. I want to build my own stuff. I don't want to sharecrop in someone's walled garden.
I also worry a great deal that OAI and Anthropic will bow to political pressure and make Claude and ChatGPT push certain political agendas, to report biased information, or refuse to help with legal requests that conflict with corporate values. I also worry about privacy and mass surveillance - chat logs are far more intimate than my search queries or selfies.
I also just don't think the open source movement has much chance of competing with the city sized data centres owned by Anthropic and OpenAI, or the hundreds of billions of dollars they have available to hire the best researchers. It costs hundreds of millions to train a frontier model, this kind of compute isn't available to the open source community.
Especially drugs- I used to think all people should have access, but overall I really wish meth just never existed and people wouldn't distribute it outside of specific circumstances. Being able to cause irreparable damage in one moment of weakness is terrible for people who have less control, and for society as a whole really.
(That's before even touching the can of worms of allowing the government to criminalize personal health choices, which feels like a glaring loophole in the Constitution to me.)
He who controls the porn controls the universe. - Baron Amodei
But it's harder thanks to US actions in the last few years, and especially in countries which can bite back.