The "Quota management outage" was such a good one that I didn't really need to read further than the title to know exactly which one you are talking about. :)
Funny how it happened before I started and this was one of the first things I learned, and I use it as example to this day.
Hah! Good that someone else can confirm that this _did_ happen :)
Do you happen to know if the specific TotT article I mentioned was published? And do you remember the details about that "financial company"? I wasn't able to find them... but the latter had to be somewhere or else we wouldn't have known about it. (But maybe I'm misremembering their outage and it was something else.)
> Do you happen to know if the specific TotT article I mentioned was published?
Yes, although if I recall it was less about "new types" and more about "don't use the same values in tests".
> And do you remember the details about that "financial company"
Can't recall.
However something recent rings a bell - but can't remember what exactly. I remember reading discussion about an outage where some people questioned "how could have X and Y swapped?", theorizing things like memory corruption/etc. I remember thinking "I think it was simpler and I think I know why"
You may be right about the main issue being about using the same data in tests. I'm pretty sure I learned about "new type" back then, but I could well be misremembering. In part because the trivial implementation of "new type" in Python wouldn't be very helpful. Maybe what we did was wrap both disk+files quotas in a struct with named fields?
The (well, I guess "a" because there might have been several) memory corruption one was about a bit flip when accessing a vtable. Oh boy, that was a nice one too. I guess I have another short story to write and won't spoil it here! :)
> In part because the trivial implementation of "new type" in Python wouldn't be very helpful.
Yeah, the "new type" only became helpful / convenient with the adoption of pytype / other type checkers, and only in heavily-annotated code. I don't know what the fix specifically was back then, some sort of struct would make sense, especially given it all ends up being a proto.
The code has been evolving a lot and I know later it had a stronger concept of a "quota unit" object (which had its own disadvantages) but it encompassed all fields in a structure that was passed along.
The "Quota management outage" was such a good one that I didn't really need to read further than the title to know exactly which one you are talking about. :)
Funny how it happened before I started and this was one of the first things I learned, and I use it as example to this day.
Hah! Good that someone else can confirm that this _did_ happen :)
Do you happen to know if the specific TotT article I mentioned was published? And do you remember the details about that "financial company"? I wasn't able to find them... but the latter had to be somewhere or else we wouldn't have known about it. (But maybe I'm misremembering their outage and it was something else.)
> Do you happen to know if the specific TotT article I mentioned was published?
Yes, although if I recall it was less about "new types" and more about "don't use the same values in tests".
> And do you remember the details about that "financial company"
Can't recall.
However something recent rings a bell - but can't remember what exactly. I remember reading discussion about an outage where some people questioned "how could have X and Y swapped?", theorizing things like memory corruption/etc. I remember thinking "I think it was simpler and I think I know why"
You may be right about the main issue being about using the same data in tests. I'm pretty sure I learned about "new type" back then, but I could well be misremembering. In part because the trivial implementation of "new type" in Python wouldn't be very helpful. Maybe what we did was wrap both disk+files quotas in a struct with named fields?
The (well, I guess "a" because there might have been several) memory corruption one was about a bit flip when accessing a vtable. Oh boy, that was a nice one too. I guess I have another short story to write and won't spoil it here! :)
> In part because the trivial implementation of "new type" in Python wouldn't be very helpful.
Yeah, the "new type" only became helpful / convenient with the adoption of pytype / other type checkers, and only in heavily-annotated code. I don't know what the fix specifically was back then, some sort of struct would make sense, especially given it all ends up being a proto.
The code has been evolving a lot and I know later it had a stronger concept of a "quota unit" object (which had its own disadvantages) but it encompassed all fields in a structure that was passed along.