brandur / sorg Goto Github PK

In sorg/content/articles/text.md you talk about why you now use varchar rather than text. But changing lengths of varchar (eg shortening) will still cause issues. Why not just use text field but with a check constraint for ensuring the current lengths on input as these can be changed at any point in time with no issue...

what tool do you use to create images such as this one? https://github.com/brandur/sorg/blob/master/content/images/redis-streams/streaming-model.svg

As heinrichhartman stated on HN, your websites is one of the few programming-related blogs which has managed to be beautifully typeset. I, and probably some others with me, would like to re-use some of the css used to obtain this look.

While this repository is quite helpful in order to understand how the final product is created, the lack of a licence means that nobody is allowed to re-use any part of it. Would you consider releasing (some part of) this repository under an open-source licence?

Hey first of all thank you for this great blog post here https://brandur.org/postgres-atomicity. I have a quick question about the visibility section. From what you wrote the HeapTupleSatisfiesVisibility method seems like it's just checking if the tuple's xmin is contained in the commit log. Am I right in assuming that it only does that if the following conditions apply ?

1. the tuple's xmin is greater than or equal to the snapshot's xmin
2. the tuple's xmin is not conainted in the snapshot's xip array
3. the tuple's xmin is smaller than the snapshot's xmax

In your example:

I would have thought that you would only need consult the commit log if a tuple has the transaction ID 126 as its xmin. In all other cases visibility could also be computed without checking if the transaction was commited. Did I misunderstand something here ?

Hi,

you have great content on your blog. I especially like (and learn from) your articles related to databases.

I have a question related to the article: https://brandur.org/http-transactions
Is there a resource you'd recommend to learn more about using DB transactions and isolation levels in (web) applications?

I've been reading about isolation levels lately. But it's all theoretical - just showing DB features. Meaningful examples, how to apply those DB features to applications, are scarce.
If you have a resource (blog, book, something else) to recommend on this topic in the vein of your article linked above I'd be grateful!

Thanks

Basically what the title says. Not necessarily a "major usability issue", but I noticed this small omission and thought I'd let you know.

Here's an image for reference:

Reading your article https://brandur.org/go-worker-pool, its unclear to me how a Task in your pool implementation should return a computed value. I have seen other implementations where the Task signature is a func returning an interface{}, error. I guess in your solution that you expect the Task func to pass in an address?

Can you elaborate your example to show how this should be handled?

tasks := []*Task{
    NewTask(func() error { return nil }),
    NewTask(func() error { return nil }),
    NewTask(func() error { return nil }),
}

p := pool.NewPool(tasks, conf.Concurrency)
p.Run()

var numErrors int
for _, task := range p.Tasks {
    if task.Err != nil {
        log.Error(task.Err)
        numErrors++
    }
    if numErrors >= 10 {
        log.Error("Too many errors.")
        break
    }
}

For e.g. if each Task creates a Result and these are stored in a

var results []Result

Apologies if this question doesn't belong here.

As an active user of Stripe, I was always curious how Stripe handles their API versioning.
Recently, I found this blog post that you wrote (https://stripe.com/fr-be/blog/api-versioning)
I want to try to implement it on one of my projects but I am missing some stuff.

Therefore, I would like to know if you can disclose more information about versioning

Thanks

Hi,

I think you've forgotten to close the response body in this article.

Here is the output from a pprof before closing the body:

Here is the output from a pprof after closing the body:

I ran 1000 requests.

• https://github.com/brandur/sorg/blob/master/content/articles/http-transactions.md

We'd like to translate the article into Japanese and publish on our tech blog https://techracho.bpsinc.jp/ if you're OK.
We make sure to indicate the link to original, title, author in the case.
I hope You'd permit us to do that.
Best regards,

Using Firefox and Chrome on Android, I'm unable to horizontally scroll code blocks.

Hi, this is regarding the "A simple HTTP retry and backoff loop in Go" article here. The code snippet currently has

func getURLData(url string) (*http.Response, []byte, error) {
	resp, err := http.Get(url)
	if err != nil {
		return nil, nil, err
	}
	defer body.Close()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return nil, nil, err
	}

	return resp, body, nil
}

But I think that the defer should be called on resp.Close(), right? The body var is not declared yet at that point (and besides, once it's initialized, it's the byte array). I'm not a Golang expert so I could be wrong here.

Hi, looks like the heading image for the Tenet Fragment had not been uploaded on GitHub/your CDN 🙂

should have releases similar to Hugo:

https://github.com/gohugoio/hugo/releases

good article, thanks for sharing!

I came across your article when I was dealing with another downside with soft deletion which is the potential for lock contention or deadlocks. If an index is created on the deleted_at column, it may cause seemingly narrow queries to hold locks on all the active rows, like any query that specifies WHERE deleted_at IS NULL. This makes any query interested in an active row contend with any other query interested in an active row, even if those queries aren't expected to return the same rows.

I liked your https://brandur.org/interfaces article and I think it could go even down the road of the "text oriendted user interfaces".

Because that's what it boils down to in the end. By the way I think that what you really meant was not "terminals" but "shells". Emacs Eshell and M-x shell show the clear distinction between the two.

I wrote about it on Reddit a year ago: https://www.reddit.com/r/emacs/comments/6y3q4k/yes_eshell_is_my_main_shell/. You might like it :)

Through https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme

brandur/json_schema#68

Hello Brandur,

I notice your posts have — instead of —.

• https://brandur.org/fragments/aws-static-hosting

• https://brandur.org/fragments/2015-update

Wondering if you need to configure special chars on S3.

Cheers,
Juanito

the biggest problem that I run in to while processing logs in name-value pairs is that people repeat the same name in multiple places in the log and count on context/ordering to figure out what is related to what.

any logfmt library should produce at least a warning if the same name is used multiple times.

I would also suggest that when including related pieces of info, a hierarchy be used (and it would be good to define a 'standard' deliminator to use, '.' and '!' are common, but a 'standardized' recommendation is a good idea.

for example
source.hostname source.ip source.port are clearly related and distinct from dest.hostname dest.ip dest.port

'.' as deliminator has the advantage that it can trivially be ignored by pretty much everything, '!' as deliminter has the advantage that it's extremely unlikely to be used by accident. both are in reasonably common use by different programs.

I'm not submitting this as a PR because the issue is a strategic/policy question, not a simple mistake, but if you would like to to do so, I will.

Hi Brandur,

I have just read about your wonderful articles on making HTTP requests transactional and idempotent. I have two questions on this topic and hope to hear your thoughts.

1. Isn’t it costly to use a single serializable transaction for the whole API?

I see that each transaction is explicitly set to use the serializable isolation level in the sample code.
AFAIK, serializable requires much more coordination cost than weaker isolation levels, such as snapshot isolation. And almost all RDBMS use weaker isolation level as default, in my opinion, mostly due to the performance cost. I understand that weaker isolation is hard to get right. But if applications need the performance, what are the options for them?

2. When dealing with foreign state mutations, does it matter that chopping the API into several transactions would make intermediate states visible to other APIs?

The intermediate states can be leaked as soon as a transaction commits, which may lead to unexpected app behaviors. For example, API A has its intermediate states exposed to API B, and API B considers some actions that are already taken. However, if API A then fails halfway through and the client never retries, then the assumed action will never complete, and API B might be doing something it shouldn’t do.

Please let me know your takes. Thanks!

Hi @brandur,

I don't quite understand why this is, but I'm cargo culting for now. – main.js

The DOMContentLoaded event is fired when the document has been completely loaded and parsed. Stylesheets, images, and subframes may not be finished loading then. If the load event is fired the page will be fully-loaded and Retina.js can be sure to find some elements with the referenced data-Attribute.

Hi, I read the article PG advisory locks in Go with built-in hashes and it is really insightful.

Using the FNV hash is brilliant way to hash string into integer.

However, I found that the advisory lock only accept pair of integer, or single bigint ¹, and in golang, it is int32 and int64 respectively, not the unsigned version.

The following is still valid:

SELECT pg_advisory_xact_lock(-2147483648, 2147483647);
SELECT pg_advisory_xact_lock(-9223372036854775808);
SELECT pg_advisory_xact_lock(9223372036854775807);

Attempting to use math.MaxUint64 might lead to errors:

select pg_advisory_xact_lock(18446744073709551615);
ERROR:  function pg_advisory_xact_lock(numeric) does not exist
LINE 1: select pg_advisory_xact_lock(18446744073709551615);
               ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

I think the output of the Hash32/Hash64 could be shifted to the int32/int64 equivalent:

func Uint64ToInt64(u64 uint64) int64 {
	if u64 > uint64(math.MaxInt64) {
		return int64(u64 - uint64(math.MaxInt64) - 1)
	}

	return int64(u64) - math.MaxInt64 - 1
}

func Uint32ToInt32(u32 uint32) int32 {
	if u32 > uint32(math.MaxInt32) {
		return int32(u32 - uint32(math.MaxInt32) - 1)
	}

	return int32(u32) - math.MaxInt32 - 1
}

And some dumb tests just to verify the boundary:

func TestUint32ToInt32(t *testing.T) {
	assert := assert.New(t)
	assert.Equal(int32(math.MaxInt32), lock.Uint32ToInt32(math.MaxUint32))
	assert.Equal(int32(math.MinInt32), lock.Uint32ToInt32(0))
}

func TestUint64ToInt64(t *testing.T) {
	assert := assert.New(t)
	assert.Equal(int64(math.MaxInt64), lock.Uint64ToInt64(math.MaxUint64))
	assert.Equal(int64(math.MinInt64), lock.Uint64ToInt64(0))
}

Cheers.

Apologies if this question doesn't belong here.

I'm not sure I understand the need to create a staging table for the jobs. Why not just wait for the user record transaction to succeed then enqueue the job?

REF: https://github.com/brandur/sorg/blob/master/content/articles/http-transactions.md#background-jobs-background-jobs

Any clarification is appreciated.

Thanks

I know this is slightly random but what tool did you use to generate the charts on https://brandur.org/http-transactions

They look really clean and a bit retro / ascii, I've not seen anything like that before...

Hi, I was binge-reading casually browsing through your website, and I noticed that at the end of the about page, the link to the Wikipedia page for Helvetica is broken, probably due to an incorrect parsing of the parentheses in the URL.

Congrats for your work by the way 🙂. I love the content of your blog and newsletters, and your approach to technological challenges in general!

it can be done!!!! All you need is '-channel_layout 5.1' for some reason, if ffmpeg sees a source as ' 5.1(side)', the resulting mp4 will not play on Apple Quicktime. But by changing it to 5.1, it works.

I couldn't just make a commit to your md

Hey, thanks a lot for writing https://brandur.org/job-drain 💛 I keep coming to the article every now and then. Based on your suggestions I implemented https://github.com/adamniedzielski/sidekiq-staged_push. Would you consider mentioning the gem at the end of the article?

In Feature Casualties of Large Databases you write:

A communication protocol that allows the query to signal out-of-band with a query’s results that it didn’t run particularly efficiently, say that it got results but wasn’t able to make use of an index. This would allow a test suite to fail early by signaling the problem to a developer instead of finding out about it in production.

But is this really necessary? I mean, if it is used only inside test suite, then you could instrument your tests so that for every query EXPLAIN is first called on it, asserting which indices it will use? Or you assert that only indices are used. So then your test assertions fail like any other assertions if it changes (like because query is changed in the future)?

So I think this is mostly a tooling issue where test frameworks does not expose such asserts readily available.

Builds are relatively fast at ~300 to 500 ms depending on load and speed of a machine, but it would be obviously more optimal if this number was an order of magnitude lower.

We do some very basic content selection to speed things up, but this could be hugely improved by being made file-specific so that only a file that fswatch detected changed would be rebuilt. This technique obviously introduces some more fragility in that we need to understand the dependencies of each output file, so whether it gets implemented should be contingent on how reliable it turns out to be.

Hi @brandur,

Many thanks for your article on idempotence, I used it to build idempotent orders and payments in my own system.

However, I ran into a problem that I am not able to wrap my head around. I would really appreciate if you could provide your take on the issue!

In the article you wrote:

Usually clients are willing to [push indeterminate requests to completion] because they want to see their requests go through, but there can be cases where a client starts working, never quite finishes, and drops forever.

So here's an example situation, if I understood you correctly:

1. Server creates an order in the DB with status PAYMENT_PENDING.
2. Server calls Stripe to execute the payment and it resolves successfully.
3. Server fails to update the order to status PAID because there's some problem with the DB or something of the sort.

I update the idempotence record after each step so the whole thing is safe to retry, but the user decides not to.

At this point I have a situation where the user has paid but the order is not placed, and they are not even aware that they actually paid (unless they check their bank statement, that is).

This is where I get confused. If I tell the user that there was an error, most people ([citation needed]) would rightfully expect that they were not charged and would deem it safe to abandon the operation. Furthermore, a lot of people ([citation needed]) would consider an abandoned operation to be equivalent to a cancelled one. So a completer pushing such orders to completion would be unexpected.

Have I misunderstood something? Are you aware of any alternative approaches?

• Perhaps, if I have faith in the completer, I should let the user know that "order is processing" and will be updated later, instead of showing an error?
• Alternatively, my completer could attempt to revert any ambiguous operations. However, since I failed to save any reference to the payment, it might now always be feasible (e. g. Stripe does not seem to provide a practical way to lookup payments using metadata).

What's your take?

Hello, i asked on Twitter (but probably wasn't the best forum for doing so). In your post, you mention gRPC's streaming as a possible alternative to webhooks. I wasn't quite sure what you meant by that. Would it possible for you to explain that a bit more?

Thanks for the post and I find all your articles very insightful! 👍

If I want to adopt your beautiful theme for hugo, where would I start?