jmholla

Born on May 30, 2020•1446 Karma

5 months ago

As far as I can tell, requiring valid ID would lose a provider safe harbor protection as it is not one of the required elements:

(3) Elements of notification.-

(A) To be effective under this subsection, a notification of claimed infringement must be a written communication provided to the designated agent of a service provider that includes substantially the following:

(i) A physical or electronic signature of a person authorized to act on behalf of the owner of an exclusive right that is allegedly infringed.

(ii) Identification of the copyrighted work claimed to have been infringed, or, if multiple copyrighted works at a single online site are covered by a single notification, a representative list of such works at that site.

(iii) Identification of the material that is claimed to be infringing or to be the subject of infringing activity and that is to be removed or access to which is to be disabled, and information reasonably sufficient to permit the service provider to locate the material.

(iv) Information reasonably sufficient to permit the service provider to contact the complaining party, such as an address, telephone number, and, if available, an electronic mail address at which the complaining party may be contacted.

(v) A statement that the complaining party has a good faith belief that use of the material in the manner complained of is not authorized by the copyright owner, its agent, or the law.

(vi) A statement that the information in the notification is accurate, and under penalty of perjury, that the complaining party is authorized to act on behalf of the owner of an exclusive right that is allegedly infringed.

truelson•

5 months ago

•on: Google broke my heart

I'm going to channel a little bit of @patio11 .

First, worth reading this on how he deals with credit agencies and debt collectors: https://www.kalzumeus.com/2017/09/09/identity-theft-credit-r... . There's gold in here for dealing with big globo-corp and how to get their attention.

Ask Google for a certified mail address so you can send them the timeline of events that occurred. This is the shibboleth that lets them know you mean business and that by not responding, they may be facing legal action. DO NOT THREATEN or mention legal action. The managerial class doesn't act that way. Just signal you are building a case against them. Start with getting that certified mailing address... you may be surprised how they respond after just that request.

If they don't respond, keep following up. Send them a timeline of events, proof of ownership even if they do not ask you what you need to prove ownership. Make it clear what this is costing you.

But here's the thing, EVERY TIME I HAVE ASKED FOR A CERTIFIED MAIL ADDRESS, the globocorp gave me what i wanted, and I never had to follow up. Every time. They don't want to deal with actual legal action from "people who know what they are doing."

It's a shibboleth. Like "Baa-ram-ewe." Use it wisely and honestly.

pera•

6 months ago

•on: Tell HN: It's now impossible to disable all AI fea...

That's strange, which OS? I am on Arch and also on 145 and I get the "Ask an AI Chatbot" in the context menu. The settings used to work in the past so I am not sure what's going on.

I believe these are all the settings I have disabled for AI:

browser.ml.chat.enabled

browser.ml.chat.menu

browser.ml.chat.page

browser.ml.chat.page.footerBadge

browser.ml.chat.page.menuBadge

browser.ml.chat.shortcuts

browser.ml.chat.sidebar

browser.ml.enable

browser.ml.linkPreview.enabled

browser.ml.pageAssist.enabled

browser.tabs.groups.smart.enabled

browser.tabs.groups.smart.userEnable

browser.tabs.groups.smart.userEnabled

extensions.ml.enabled

sidebar.notification.badge.aichat

Am I missing anything?

opan•

7 months ago

I would say the X1 Carbon is not a real ThinkPad, and anything with Yoga in the name even less so. His brother's P series should be fine, though. Stick to X (but not X1), T, W, P series, and note that T is the "normal one". Also avoid s variants, e.g. T14s is not the same beast as a T14. Once you've filtered to this point you should rule out a lot of problems with build quality or soldered parts, though even then they don't really make 'em like they used to. Speaking of soldered parts, there was a bit of a dark age where even most of the normal ThinkPads had half or fully soldered RAM, but they're just starting to come back from that. T14 gen 5 is unsoldered, but gens 1 through 3 had soldered RAM, IIRC. Wikipedia has a table you can check for this. So, sadly the used market is gonna be full of the soldered models for a while.

embedding-shape•

7 months ago

•on: Yt-dlp: External JavaScript runtime now required f...

Seems its already in Arch's repositories, and seems to work, just add another flag to the invocation:

    yt-dlp --cookies-from-browser firefox --remote-components ejs:github -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best" 'https://www.youtube.com/watch?v=XXX'

It is downloading a solver at runtime, took maybe half a second in total, downloads are starting way faster than before it seems to me.

    [youtube] [jsc:deno] Solving JS challenges using deno
    [youtube] [jsc:deno] Downloading challenge solver lib script from  https://github.com/yt-dlp/ejs/releases/download/0.3.1/yt.solver.lib.min.js

It would be great if we could download the solver manually with a separate command, before running the download command, as I'm probably not alone in running yt-dlp in a restricted environment, and being able to package it up together with the solver before runtime would let me avoid lessening the restrictions for that environment. Not a huge issue though, happy in general the start of downloads seems much faster now.

Uehreka•

9 months ago

•on: VibeVoice: A Frontier Open-Source Text-to-Speech M...

It’s tough to name the best local TTS since they all seem to trade off on quality and features and none of them are as good as ElevenLabs’ closed-source offering.

However Kokoro-82M is an absolute triumph in the small model space. It curbstomps models 10-20x its size in terms of quality while also being runnable on like, a Raspberry Pi. It’s the kind of thing I’m surprised even exists. Its downside is that it isn’t super expressive, but the af_heart voice is extremely clean, and Kokoro is way more reliable than other TTS models: It doesn’t have the common failure mode where you occasionally have a couple extra syllables thrown in because you picked a bad seed.

If you want something that can do convincing voice acting, either pay for ElevenLabs or keep waiting. If you’re trying to build a local AI assistant, Kokoro is perfect, just use that and check the space again in like 6 months to see if something’s beaten it. https://huggingface.co/hexgrad/Kokoro-82M

qudat•

11 months ago

•on: Dumb Pipe

At pico.sh we built something similar but using SSH: https://pipe.pico.sh

lpeancovschi•

11 months ago

•on: Programming vehicles in games

A while ago I made a game for iOS that simulates vehicles and drifting. It was built in SpriteKit but very easy to do in any 2d game engine. The idea is: you put two wheels in front, connect them to the rectangle car shape with pin joints and then you apply force to the wheels. The angle of the force is very easy to calculate: x = force * cos(bodyRotation + wheelRotation) y = force * sin(bodyRotation + wheelRotation)

That's it! I also added skid particles. The drifting was achieved by playing around with the wheels and body damping. The game is here: https://apps.apple.com/app/drift-mania-infinite-car-racer/id...

gopalv•

11 months ago

•on: Uv: Running a script with dependencies

This is my absolute favourite uv features and the reason I switched to uv.

I have a bunch of scripts in my git-hooks which have dependencies which I don't want in my main venv.

#!/usr/bin/env -S uv run --script --python 3.13

This single feature meant that I could use the dependencies without making its own venv, but just include "brew install uv" as instructions to the devs.

rwiggins•

11 months ago

•on: Jqfmt like gofmt, but for jq

Oh, fantastic. jq has become an integral part of work for me.

I'll use this opportunity to plug the one-liner I use all the time, which summarizes the "structure" of a doc in a jq-able way: https://github.com/stedolan/jq/issues/243#issuecomment-48470... (I didn't write it, I'm just a happy user)

For example:

    $ curl -s 'https://ip-ranges.amazonaws.com/ip-ranges.json' | jq -r '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]'
    .
    .createDate
    .ipv6_prefixes
    .ipv6_prefixes[]
    .ipv6_prefixes[].ipv6_prefix
    .ipv6_prefixes[].network_border_group
    .ipv6_prefixes[].region
    .ipv6_prefixes[].service
    .prefixes
    .prefixes[]
    .prefixes[].ip_prefix
    .prefixes[].network_border_group
    .prefixes[].region
    .prefixes[].service
    .syncToken

(except I have it aliased to "jq-structure" locally of course. also, if there's a new fancy way to do this, I'm all ears; I've been using this alias for like... almost a decade now :/)

In the spirit of trying out jqfmt, let's see how it formats that one-liner...

    ~  echo '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]' | ~/go/bin/jqfmt -ob -ar -op pipe
    [
        path(..) | 
        map(if type == "number" then "[]" else tostring end) | 
        join(".") | 
        split(".[]") | 
        join("[]")
    ] | 
        unique | 
        map("." + .) | 
        .[]%
    ~

Not bad! Shame that jqfmt doesn't output a newline at the end, though. The errant `%` is zsh's partial line marker. Also, `-ob -ar -op pipe` seems like a pretty good set of defaults to me - I would prefer that over it (seemingly?) not doing anything with no flags. (At least for this sample snippet.)

tptacek•

on June 24, 2025

•on: Excalidraw+ Is Now SoC 2 Certified

This is all good, just a note for anybody reading this to the end: there's basically no way not to pass your Type 1, at least not if you're using a serious auditor. The point of a Type 1 is to document a point-in-time baseline. The Type 2 is the first "real" audit, and basically just checks whether you reliably did all the things you attested to in your Type 1.

All that is to say: you want to minimize the amount of security work you do for your Type 1, down to a small set of best practices you know you're going to comply with forever (single sign-on and protected branches are basically 90% of it). You can always add controls later. Removing them is a giant pain in the ass.

This is always my concern for people going into SOC2 cold: vendors in the space will use the Type 1 as an opportunity for you to upskill your team and get all sorts of stuff deployed. A terrible and easily avoided mistake.

I write this only because the piece ends with Excalidraw psyched to have cleared their Type 1. I hope their auditors told them they were always going to clear that bar.

shakna•

on June 22, 2025

•on: Unexpected security footguns in Go's parsers

Tangent for breaking Python's JSON parser: This has worked for five years. The docs do not say that parsing an invalid piece will result in a RecursionError. They specify JSONDecodeError and UnicodeDecodeError. (There is a RecursionError reference to a key that is off by default - but if its off, we can still raise this...)

    #!/bin/sh

    # Python will hit it's recursion limit
    # If you supply just 4 less than the recursion limit
    # I assume this means there's a few objects on the call stack first
    # Probably: __main__, print, json.loads, and input.

    n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')"

    echo "N: $n"

    # Obviously invalid, but unparseable without matching pair
    # JSON's grammar is... Not good at being partially parsed.
    left="$(yes [ | head -n "$n" | tr -d '\n')"

    # Rather than exploding with the expected decodeError
    # This will explode with a RecursionError
    # Which naturally thrashes the memory cache.
    echo "$left" | python3 -c 'import json; print(json.loads(input()))'

oftenwrong•

on June 22, 2025

•on: Git Notes: Git's coolest, most unloved feature (2...

Another little-known feature is git trailers:

https://alchemists.io/articles/git_trailers

These are key-value structures data that can be included on a commit when it is created. These are used by some systems for attaching metadata. For example, Gerrit uses this for attaching its Change-Id.

Ayesh•

on May 30, 2025

•on: Show HN: Git-Add–Interactive with Enhancements

Congratulations on publishing this. I use `git add -p` quite a lot, and this project looks interesting!

I knew that you could place a `git-xyz` executable and you can call it as `git xyz`. I didn't know you could do it with flags !?!

A small video or some screenshots would help a lot. If you can record interactivity with ascii-cinema, that will be even better.

majke•

on May 26, 2025

•on: TIL: timeout in Bash scripts

My fav little-known trick is to test various syscalls fail with strace fault injection, like:

  $ strace -e trace=clone -e fault=clone:error=EAGAIN

random link: https://medium.com/@manav503/using-strace-to-perform-fault-i...

uallo•

on May 12, 2025

•on: Just use HTML

You may be able to debug it with https://github.com/mozilla/readability

WarOnPrivacy•

on May 4, 2025

•on: Why is Windows consistently losing market share? (...

It'd be easier to evangelize for Windows if I didn't constantly have to weed out anti-user behaviors.

Below are cmd's I run for new Win11 setups (foistware uninstalls are a another page of PS commands).

    :: Disable Web from Taskbar Search
    reg add HKCU\Software\Microsoft\Windows\CurrentVersion\Search /v BingSearchEnabled /t REG_DWORD /d 0 /f
    reg add HKCU\Software\Policies\Microsoft\Windows /v DisableSearchBoxSuggestions /t REG_DWORD /d 1 /f
    
    :: Turn Off MS Account Notifications in Start (reduces MS Account grooming)
    reg add "HKCU\Software\Policies\Microsoft\Windows\CurrentVersion\AccountNotifications" /v "DisableAccountNotifications" /t REG_DWORD /d "1" /f
    
    :: Win11 Start Menu to the left side
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced" /v "TaskbarAl" /t REG_DWORD /d "0" /f
    
    :: Removes Task View from the Taskbar
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced" /v "ShowTaskViewButton" /t REG_DWORD /d "0" /f
    
    :: Set to show file assocations
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced" /v "HideFileExt" /t REG_DWORD /d "0" /f
    
    :: Removes Widgets from the Taskbar
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced" /v "TaskbarDa" /t REG_DWORD /d "0" /f
    
      :: Win11 Turn Off Copilot
    reg add "HKCU\Software\Policies\Microsoft\Windows\Copilot" /v "TurnOffWindowsCopilot" /t REG_DWORD /d "1" /f
    
    :: P&G Disable Advertising ID for Personalized Ads
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\AdvertisingInfo" /v "Enabled" /t REG_DWORD /d "0" /f
    reg add "HKLM\Software\Microsoft\Windows\CurrentVersion\AdvertisingInfo" /v "DisabledByGroupPolicy" /t REG_DWORD /d "1" /f
    
    :: P&G Disable Show me suggested content
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "SubscribedContent-338393Enabled" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "SubscribedContent-353694Enabled" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "SubscribedContent-353696Enabled" /t REG_DWORD /d "0" /f
    
    sc delete XboxGipSvc
    sc delete XblAuthManager
    sc delete XblGameSave
    sc delete XboxNetApiSvc
    
    :: System->Notifications->Disable notifications can play sounds,show reminders on lock, show notifications on lock
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Notifications\Settings" /v "NOC_GLOBAL_SETTING_ALLOW_NOTIFICATION_SOUND" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Notifications\Settings" /v "NOC_GLOBAL_SETTING_ALLOW_TOASTS_ABOVE_LOCK" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Notifications\Settings" /v "LockScreenToastEnabled" /t REG_DWORD /d "0" /f
    
    :: System->Notifications->Additional Settings, disable all
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "SubscribedContent-310093Enabled" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\UserProfileEngagement" /v "ScoobeSystemSettingEnabled" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "SubscribedContent-338389Enabled" /t REG_DWORD /d "0" /f
    
    :: After sleep, Never require signin 
    reg add "HKCU\Control Panel\Desktop" /v DelayLockInterval /d 0xffffffff /t REG_DWORD /f
    
    :: Show and enable Max power plan
    powercfg /s SCHEME_MIN
    
    :: Restore UsersMustLogon checkbox to controluserpasswords2
    reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\PasswordLess\Device" /v DevicePasswordLessBuildVersion /t REG_DWORD /d 00000000 /f
    
    :: Disable auto-submit samples to MSAV
    reg add "HKLM\SOFTWARE\Policies\Microsoft\Windows Defender\Spynet" /v "SubmitSamplesConsent" /t REG_DWORD /d "2" /f
    
    :: Set LockScreen to picture 
    reg add "HKCU\Control Panel\Desktop" /v "LockScreenAutoLockActive" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "RotatingLockScreenEnabled" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Lock Screen" /v "SlideshowEnabled" /t REG_DWORD /d "0" /f
    
    :: Disable Lockscreen widgets for current, all users
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\Lock Screen" /v "LockScreenWidgetsEnabled" /t REG_DWORD /d "0" /f
    reg add "HKLM\SOFTWARE\Policies\Microsoft\Dsh" /v "DisableWidgetsOnLockScreen" /t REG_DWORD /d "0" /f
    
    :: Personalize Lockscreen = picture
    :: Set Lockscreen status = None (no joy 24H2)
    :: Disable "Get fun facts, tips, tricks, and more on your lock screen"
    reg add "HKCU\Software\Policies\Microsoft\Windows\CloudContent" /v "DisableSpotlightCollectionOnDesktop" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "RotatingLockScreenOverlayEnabled" /t REG_DWORD /d "0" /f
    reg add "HKCU\Software\Microsoft\Windows\CurrentVersion\ContentDeliveryManager" /v "SubscribedContent-338387Enabled" /t REG_DWORD /d "0" /f

    
    :: Win11 Computer-> R-Click-> restore Win10 Context Menu
    reg add "HKCU\Software\Classes\CLSID\{86ca1aa0-34aa-4e8b-a509-50c905bae2a2}\InprocServer32" /ve /d ""

staplung•

on April 20, 2025

•on: Meta's Monopoly Made It a Fair-Weather Friend

Couldn't find this on Internet Archive but if you set your user agent to something that looks like google's crawler you can get past the paywall. It's easy on Safari if you already have the Develop menu enabled.

Develop > User Agent > Other...

Then use:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Note that this works for most paywalls, not just Wired.

freddie_mercury•

on April 19, 2025

•on: Cozy video games can quell stress and anxiety

There are tons of games like A Short Hike.

Alba, Little Kitty Big City, Lil Gator Game, Haven Park, Time on Frog Island, Little Wings Deliveries , The Kind Chamomile, Smushi Come Home, Petit Island, Luna's Fishing Garden, ...

haswell•

on March 9, 2025

•on: With AI you need to think bigger

I recently discovered that some of the Raspberry Pi models support the Linux Kernel's "Gadget Mode". This allows you to configure the Pi to appear as some type of device when plugged into a USB port, i.e. a Mass Storage/USB stick, Network Card, etc. Very nifty for turning a Pi Zero into various kinds of utilities.

When I realized this was possible, I wanted to set up a project that would allow me to use the Pi as a bridge from my document scanner (has the ability to scan to a USB port) to a SMB share on my network that acts as the ingest point to a Paperless-NGX instance.

Scanner -> USB "drive" > Some of my code running on the Pi > The SMB Share > Paperless.

I described my scenario in a reasonable degree of detail to Claude and asked it to write the code to glue all of this together. What it produced didn't work, but was close enough that I only needed to tweak a few things.

While none of this was particularly complex, it's a bit obscure, and would have easily taken a few days of tinkering the way I have for most of my life. Instead it took a few hours, and I finished a project.

I, too, have started to think differently about the projects I take on. Projects that were previously relegated to "I should do that some day when I actually have time to dive deeper" now feel a lot more realistic.

What will truly change the game for me is when it's reasonable to run GPT-4o level models locally.

paranoidrobot•

on March 10, 2025

•on: Zero-Downtime Kubernetes Deployments on AWS with E...

We had to figure this out the hard way, and ended up with this approach (approximately).

K8S provides two (well three, now) health checks.

How this interacts with ALB is quite important.

Liveness should always return 200 OK unless you have hit some fatal condition where your container considers itself dead and wants to be restarted.

Readiness should only return 200 OK if you are ready to serve traffic.

We configure the ALB to only point to the readiness check.

So our application lifecycle looks like this:

* Container starts

* Application loads

* Liveness begins serving 200

* Some internal health checks run and set readiness state to True

* Readiness checks now return 200

* ALB checks begin passing and so pod is added to the target group

* Pod starts getting traffic.

time passes. Eventually for some reason the pod needs to shut down.

* Kube calls the preStop hook

* PreStop sends SIGUSR1 to app and waits for N seconds.

* App handler for SIGUSR1 tells readiness hook to start failing.

* ALB health checks begin failing, and no new requests should be sent.

* ALB takes the pod out of the target group.

* PreStop hook finishes waiting and returns

* Kube sends SIGTERM

* App wraps up any remaining in-flight requests and shuts down.

This allows the app to do graceful shut down, and ensures the ALB doesn't send traffic to a pod that knows it is being shut down.

Oh, and on the Readiness check - your app can use this to (temporarily) signal that it is too busy to serve more traffic. Handy as another signal you can monitor for scaling.

e: Formatting was slightly broken.

sgarland•

on March 8, 2025

•on: Postgres Just Cracked the Top Fastest Databases fo...

> Basically we started to collected detailed analytics and thus had a rapidly growing table of around 2B records of user events during their sessions. As it grew past a 500 million records it turned out to be impossible to query this table in any thing close to real-time

Analytics isn't typically something that needs real-time capabilities, for one.

> a rapidly growing table [emphasis mine]

I think I see part of the problem here. If you had a single table, that means it's completely denormalized, so your schema probably looked something like this (or wider):

    CREATE TABLE UserEvent (
      id UUID PRIMARY KEY,
      user_id UUID NOT NULL,
      user_ip_address TEXT NOT NULL,
      user_agent TEXT NOT NULL,
      event_data JSONB,
      created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
      event_type TEXT
    );

    CREATE INDEX UserEvent_user_id_idx ON UserEvent (user_id);
    CREATE INDEX UserEvent_created_at_idx ON UserEvent (created_at);

The JSON blob might be anywhere from a few hundred bytes to well over a kilobyte, and probably duplicates data already present as a scalar, like IP address, user agent string, timestamp, etc. I'll use the middle ground and say the JSONB objects are on average 500 bytes when stored. Now, the rest.

A UUID, if stored as its native type (or BINARY(16) in MySQL - don't sleep on this, MySQL folks; it makes a huge difference at scale) is 16 bytes. That's double the size of a BIGINT, and quadruple the size of an INT4. Also, unless you're using UUIDv7 (or UUIDv1, but no one does), it's not k-sortable. Since Postgres doesn't cluster tuples around the PK [yes, I know all indices in Postgres are technically secondary] like MySQL/InnoDB does, this doesn't immediately thrash the B+tree in the same way, but it does thrash the visibility map, and it does bloat the WAL. There are various arguments for why you shouldn't use a monotonic integer as a surrogate key, but IMO they're largely overblown, and there are workarounds to not publicly disclose it.

IPv4 addresses, stored in dotted-quad as a string, are a maximum of 15 characters, storing in 16 bytes as TEXT or VARCHAR. If stored instead in the Postgres native INET type, that drops to 7 bytes, plus you get built-in validation. If you had INT4 UNSIGNED available (as MySQL does natively), you could even store them in their numeric form and save another 3 bytes, though you lose the validation.

User Agent strings are huge, usually over 100 bytes. They're also not that unique, relatively speaking. Even if you need to know the patch version of the browser, anyone with a browser doing automatic updates is going to stay more or less in sync. The point is this could easily be a lookup table, with either a SMALLINT (2^15 - 1 maximum values, or 2^16 - 1 if you use unsigned values; possible with an extension in Postgres) or an INT (2^31 -1 maximum values) as the PK.

Not going to touch on JSON objects because the things you might want to know are endless. TOAST and de-TOAST can be slow; if you need low latency, you should normalize your data.

There may or may not be extracted scalars, which can be beneficial during queries. Again, though, lookup tables (or even native ENUM types, if the values are limited) are crucial at scale.

As it stands, the table will have an average row size of 664 bytes (assuming an average of 12 bytes stored for the IP, 100 bytes stored for the UA, 500 bytes stored for the JSONB, and 12 bytes stored for the event type). That's 332 GB for 500,000,000 rows. You could shave a couple of bytes off by aligning columns [0], which saves 1 GB. If the IP addresses and UA strings were lookup tables, each with an INT4, that saves 104 bytes per row. If you made the PK for the table a BIGINT, that saves another 8 bytes per row. The total savings between column alignment and basic normalization is 114 bytes per row, or 57 GB.

This doesn't touch on the indices, either. If you're using PG 13+, you get B+tree de-duplication [1] for free, which can help with some denormalized data, but not if you have anything with high cardinality, like a timestamp, or a UUID. With lookup tables, you would of course need to index those FKs (whether or not you're enforcing constraints), which adds some size, but is still a huge net gain.

> I know I could have used some type of daily aggregation combined with a weekly aggregation, etc to roll up the data incrementally. A dev tried this and yeah, it hide the slow queries but then it became inflexible in terms of reporting. And writing and maintaining these cronjobs is a lot of work.

And shifting your entire analytics workload isn't a lot of work? Between ROLLUP [2] and MATERIALIZED VIEW [3], which can automatically refresh itself with a cron, this doesn't seem that burdensome.

> Also BigQuery bill for https://web3dsurvey.com is like $0.25 month and it is dealing with millions of records in its 3 month window of stored data.

Then you're in the free tier (<= 1 TiB/month of processed data), because after that it's $6.25/TiB. Also worth noting there is a massive difference between millions of rows and billions of rows. The former can be handled by practically any RDBMS on any hardware, with a completely unoptimized schema. The latter requires some thought if you want it to be performant.

This isn't at all to say that specialized DBs don't have their place, because they absolutely do. If you need a KV store, use a KV store, not an RDBMS. If you need OLAP, use something designed for OLAP. The difference is scale. At startup or side project scale, you can easily do everything (including pub/sub) with an RDBMS, and if you put thought into its design and usage, you can take it a lot farther than you'd think. Eventually, you may hit a point where it's counter-productive to do so, and then you should look into breaking tasks out.

The issue I see happening time and time again is devs have little to no expertise in DBs of any kind, but since everyone says "Postgres is all you need," they decide to use it for everything, except they don't know what they're doing. If you do that, yeah, you're gonna have problems fairly early on, and then you'll either throw your hands up and decide you really need a bevy of specialized DBs, caches, and message queues (which introduces a lot of complexity), or you'll vertically scale the DB. If you choose the latter, by the time you hit scaling limits, you're easily spending $25K/month on the DB alone. If you opt to hire someone with DB expertise at this point, you'll spend about that if not more in personnel costs, and not only will it take them weeks if not months to unravel everything, your devs will be constantly complaining that queries are now "too complex" because they have to do some JOINs, and they're being told to stop chucking everything into JSON. If instead, you took at most a week to learn some RDBMS basics by a. reading its manual front-to-back b. hands-on experience, trying things out you could almost certainly get much farther on much less.

[0]: https://www.enterprisedb.com/blog/rocks-and-sand

[1]: https://www.postgresql.org/docs/current/btree.html#BTREE-DED...

[2]: https://www.postgresql.org/docs/current/queries-table-expres...

[3]: https://www.postgresql.org/docs/current/rules-materializedvi...

ho_schi•

on March 1, 2025

•on: An update on Mozilla's terms of use for Firefox

They are only three viable web-engines left over from the second browser-war:

  * Blink (Google): Used in everything, from Chrome, Edge, Opera, Qt-Toolkit, Electron.
  * Gecko (Mozilla): Firefox. And Waterfox? I assume Gecko is still hard to integrate.
  * WebKit and WebKitGtk (Apple and Gtk): Safari, Epiphany and Gtk-Toolkit. Easy to integrate. And the only engine where I’m aware that actually two side actively cooperate in development.

Epiphany is small and nice, but they need a lot more developers. And I think they should use ffmpeg, gstreamer seems to be a source of issues for many years. But again, they need us, every helper capable of C++ is welcome.

Ladybird an another new engine, implemented in C++. But it is in alpha-state, only for developers. Everyone else who tries to show us a new browser means “use that Google thing with another name on it”.

Epa095•

on Feb 22, 2025

•on: I found a backdoor into my bed

FYI there is https://github.com/Hypfer/Valetudo which acts as a replacement of the cloud service. I'd you buy one of the supported devices you root it to point to the Valetudo server (which you run yourselves). You can then put the robot on its own WiFi without Internet access if you want to be even more secure.

sah2ed•

on Sept 17, 2016

•on: Elon Musk on How to Build the Future

> Amusingly if you read chapter 1 of Founders at Work quite a big part of what worked at PayPal may have been firing Elon Musk. Max Levchin largely built PayPal tech wise using Unix and then it was merged with Musk's X.com and Musk became CEO and wanted to switch everything to Windows.

I encourage you to read "Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future." Musk talked about PayPal and the Windows switch in detail in Appendix 2:

“As for the technology change, that’s not really well understood. On the face of it, it doesn’t sound like it makes much sense for us to be writing our front-end code in Microsoft C++ instead of Linux. But the reason is that the programming tools for Microsoft and a PC are actually extremely powerful. They’re developed for the gaming industry. I mean, this is going to sound like heresy in a sort of Silicon Valley context, but you can program faster, you can get functionality faster in the PC C++ world. All of the games for the Xbox are written in Microsoft C++. The same goes for games on the PC. They’re incredibly sophisticated, hard things to do, and these great tools have been developed thanks to the gaming industry. There were more smart programmers in the gaming industry than anywhere else. I’m not sure the general public understands this. It was also 2000, and there were not the huge software libraries for Linux that you would find today. Microsoft had huge support libraries. So you could get a DLL that could do anything, but you couldn’t get—you couldn’t get Linux libraries that could do anything.

“Two of the guys that left PayPal went off to Blizzard and helped created World of Warcraft. When you look at the complexity of something like that living on PCs and Microsoft C++, it’s pretty incredible. It blows away any website.

“In retrospect, I should have delayed the brand transition, and I should have spent a lot more time with Max getting him comfortable on the technology. I mean, it was a little difficult because like the Linux system Max had created was called Max Code. So Max has had quite a strong affinity for Max Code. This was a bunch of libraries that Max and his friends had done. But it just made it quite hard to develop new features. And if you look at PayPal today, I mean, part of the reason they haven’t developed any new features is because it’s quite difficult to maintain the old system.

“Ultimately, I didn’t disagree with the board’s decision in the PayPal case, in the sense that with the information that the board had I would have made maybe the same decision. I probably would have, whereas in the case of Zip2 I would not have. I thought they just simply made a terrible decision based on information they had. I don’t think the X.com board made a terrible decision based on the information they had. But it did make me want to be careful about who invested in my companies in the future.

“I’ve thought about trying to get PayPal back. I’ve just been too strung out with other things. Almost no one understands how PayPal actually worked or why it took off when other payment systems before and after it didn’t. Most of the people at PayPal don’t understand this. The reason it worked was because the cost of transactions in PayPal was lower than any other system. And the reason the cost of transactions was lower is because we were able to do an increasing percentage of our transactions as ACH, or automated clearinghouse, electronic transactions, and most importantly, internal transactions. Internal transactions were essentially fraud-free and cost us nothing. An ACH transaction costs, I don’t know, like twenty cents or something. But it was slow, so that was the bad thing. It’s dependent on the bank’s batch processing time. And then the credit card transaction was fast, but expensive in terms of the credit card processing fees and very prone to fraud. That’s the problem Square is having now.

“Square is doing the wrong version of PayPal. The critical thing is to achieve internal transactions. ...

qazxcvbnm•

on Feb 13, 2025

•on: Automating the Vim Workplace (2020)

This is very powerful. Unfortunately, as an Ex command, `!` only works on full lines. One can of course also teach vim to operate `!` over any motion (or visual selection), whether it is part of a line, a full line, or a block https://vi.stackexchange.com/a/46304/48750.

mjevans•

on Feb 12, 2025

•on: PgAssistant: OSS tool to help devs understand and ...

  __264516 Feb 12 11:44  pgassistant.gif
  22782965 Feb 12 11:46  pgassistant.gif.raw.gif
  _2120322 Feb 12 11:55  pgassistant.gif.av1-20.mp4
  __245780 Feb 12 11:56  pgassistant.gif.av1-55.mp4

wget -O pgassistant.gif.raw.gif 'https://github.com/nexsol-technologies/pgassistant/blob/main...'

ffmpeg -h encoder=libaom-av1

ffmpeg -i pgassistant.gif.raw.gif -c:v libaom-av1 -crf 20 -cpu-used 8 -movflags '+faststart' -pix_fmt yuv420p pgassistant.gif.av1-20.mp4

ffmpeg -i pgassistant.gif.raw.gif -c:v libaom-av1 -crf 55 -cpu-used 8 -movflags '+faststart' -pix_fmt yuv420p pgassistant.gif.av1-55.mp4

A place to start from at least, note the 264516 gif is what's currently on the landing page, with the wget command to grab the raw file.

Bender•

on Jan 22, 2025

•on: LWN sluggish due to DDoS onslaughts from AI-scrape...

User-agent aside there are usually small details bots leave out unless they are using headless chrome of course. Most bots can't do HTTP/2.0 yet all common browsers support it. Most bots will not be sending cors, no-cors, navigate sec_fetch_mode headers whereas browsers do. Some bots do not send a accept_language header. Those are just a few things one can look for and deal with in simple web server ACL's. Some bots do not support http-keepalive, though this can knock out some poor middle boxes if dropping connections that do not support http keepalive.

At the tcp layer some bots do not set MSS options or use very strange values. This can get into false positives so I just don't publish IPv6 records for my web servers and then limit to an MSS range of 1280 to 1460 on IPv4 which knocks out many bots.

There are always the possibilities of false positives but they can be logged and reviewed acceptable losses should the load on the servers get too high. Another mitigating control is to perform analysis on previous logs and use maps to exclude people that post on a regular basis or have logins to the site assuming none of them are part of the problem. If a registered user is part of the problem give them an error page after {n} requests.

unscaled•

on Dec 18, 2024

•on: OpenAUTH: Universal, standards-based auth provider

- JWT algorithm is not configurable: good!

- JWT Algorithm is RS512: uggh.

Huge access tokens, slow validation AND bad security at the same time, boy we must be lucky.

- Encrypted JWT when saving sensitive information in cookie? Good start...

- ... Using Asymmetric encryption? Oh no...

- RSA-OAEP: At least it's not PKCS#1.5 amirite?

- Same RSA key is used for encryption and signature(?) Not great.

- Stateful Access Tokens: well...

I'm not sure how I feel about using stateful access tokens here at all. Since there is already a KV dependency, there are some advantages to storing stateful access tokens in the KV, most importantly you can easily revoke the tokens directly by deleting them. Revoking stateless tokens, on the other hand, is quite hard and not something that most web applications would care to implement.

The most common trade-off (and indeed, half of the raison d'être for OAuth 2.0 refresh tokens) is to have a very short-lived (e.g. 5 minutes) stateless access token and a long-lived stateful refresh token (which OpenAUTH already does). Revocation would still come with some delay, but you won't be able to use an outdated token for a long time after the user logs out or changes password. This is an acceptable trade-off for many applications, but I'm not sure if it's right to offer it as the only solution in a software package that wants to be generic: many applications will have compliance requirements that could rule out accepting logged out session tokens for such a period.

- JWT in any form or manner

The elephant in the room. Since JWT allows you to choose your own algorithm and offers some some rather bad options, using JWT can be considered as "Rolling your own crypto Lite". You have a lot of choices here and if you are not an expert you're bound to make some wrong choices: such as the ones I've listed above. If OpenAUTH had used PASETO for its tokens, it wouldn't be running into these issues at least since no clearly insecure options are available.

If you do use JWT, for the love of all that is good, never stray away from this path:

1. For symmetric tokens, use the HS* family of algorithms. That's the only available option anyway.

2. When using HS256/384/512, you should use randomly generated secrets from /dev/urandom [1]. The secret size in bits should be the same size as the HS algorithm bits (i.e. 32 bytes for HS256, 48 bytes for HS384 and 64 bytes for HS512). In any case, you should NEVER use passwords as the HS256/384/512 algorithm secret.

3. Do not use asymmetric tokens unless the there are are multiple token verifying parties which are separate from the token issuer. If the token issuer is the same as the verifier, you should use a symmetric token. If you've got one issuer and one verifier, you should probably still use a symmetric token with a shared secret, since there is no issue of trust. Asymmetric cryptography is always an order of magnitude easier to screw up than symmetric cryptography.

4. If you do use asymmetric cryptography, always use Ed25519. If you are forced to use something else for compatibility, use ES256/384/512. It still has some issues (especially if your random number generator is unreliable), but it's still better than RSA. You really want to use Ed25519 though.

5. If you want to encrypt JWT, don't. JWE is too hard to use correctly, and the only reliably secure option that is usually implemented by libraries (A256GCMKW) is slow and it's not very popular so I'm not sure how much analysis the algorithm (AES Key Wrap) has seen.

6. The best and easiest option for encryption if you really must use JWT (and can't use PASETO): Just take your payload, encrypt it with NaCl/Libsodium (secretbox[2]), base64 encode the result and stuff it inside a JWT claim. This will be faster, easier and more secure than anything JWE can offer.

[1] https://www.latacora.com/blog/2018/04/03/cryptographic-right...

[2] https://libsodium.gitbook.io/doc/secret-key_cryptography/sec...

adrian_b•

on Jan 13, 2025

•on: Why is my CPU usage always 100%?

Direct reading without the risk of reading incorrect values is possible only when the timer is implemented using a synchronous counter instead of an asynchronous counter and the synchronous counter must be fast enough to ensure a stable correct value by the time when it is read, and the reading signal must be synchronized with the timer clock signal.

Synchronous counters are more expensive in die area than asynchronous counters, especially at high clock frequencies. Moreover, it may be difficult to also synchronize the reading signal with the timer clock. Therefore the second solution may be preferable, which uses a separate capture register for reading the timer value.

This was implemented in the timer described in TFA, but it was done in a wrong way.

The capture register must either ensure that the capture is already complete by the time when it is possible to read its value after giving a capture command, or it must have some extra bit that indicates when its value is valid.

In this case, one can read the capture register until the valid bit is on, having a complete certainty that the end value is correct.

When adding some arbitrary delay between the capture command and reading the capture register, you can never be certain that the delay value is good.

Even when the chosen delay is 100% effective during testing, it can result in failures on other computers or when the ambient temperature is different.