Category Archives: programming

Enterprise / Server Cloud Storage (not “Personal” cloud storage)

02 Saturday Jan 2021

Posted by in aide-memoire, programming, tech

≈ Leave a comment

Cheapest:

  • Backblaze B2: $5/Tb/mo (data transfer: $10/Tb)
  • Wasabi Storage: $5/Tb/mo (data transfer: ??)
  • OVH physical server: $3/Tb/mo (zero data transfer and other charges, but you do need to setup ZFS and some API layer software yourself).

Backblaze and Wasabi are both S3 compatible object stores. Note that data transfer can easily dominate storage charges, for even somewhat frequently accessed data. OVH is therefore much cheaper than it appears.

Other Options

  • google / gcloud: $10/Tb/mo
    • 1 Tb  => $10/mo ($120/pa)
    • Nearline (USA): 1000 * 0.01 => $10/mo (Using prices for nearline storage, South Carolina data center)
    • Nearline (Mumbai): 1000 * $0.016 = $16/mo ($200/pa)
    • Nearline has some charges if stored for less than 30d (min term). So could be a bit more than $10 (perhaps the same as S3’s intelligent tiering)
  • amazon / s3
    • 1 Tb => $12.5/mo  (between $12.5 and $20 because intelligent tiering will take some time to kick in).
    • Using prices for S3 Intelligent Tiering, USA data center: 1000 * 0.0125 => $12.5
  • Digital Ocean / volumes: (Block storage built on Ceph)
    • 1 Tb => $100/mo (approx 10x S3 prices, yiiikes.)
    • rsync’able storage
  • Digital Ocean / spaces: $0.02 gb/mo (S3 Compatible File Storage)
    • 1 Tb => $20/mo (about the same as S3 without intelligent tiering)
  • Backblaze B2: $5/Tb/mo
    • $10/Tb data transfer; cheaper than Amazon, but can still dominate.
    • S3 compatible API

Max value

Store HOT on Digital Ocean spaces (save on bandwidth)

Move older items to gcloud nearline or lower tier

#[[Hosting / Servers / Cloud Guide]]

Advertisement

“Free” SQL Databases

29 Tuesday Dec 2020

Posted by in aide-memoire, programming, tech

≈ Leave a comment

  • Heroku Postgres
    • Good & simple
  • IBM DB2: No credit card required, re-activate by email every 90 days
    • 200 MB of data storage
    • 15 simultaneous connections
    • Shared multitenant system
  • “Always free” Oracle Cloud instance (ptooie, slow as molasses) + apt install postgres
  • Supabase: Postgres, realtime, kinda like Firebase but with SQL and a real postgres instance. Free while in beta
  • Recommendations:

Timeseries Databases

20 Sunday Dec 2020

Posted by in aide-memoire, programming

≈ Leave a comment

Tags

,

tl;dr use Timescale (open source extensions to postgres) – All the power, expressivity and ease of SQL w/horizontal scalability via partitioning. Like your own, Amazon RedShift. Faster and obvs. much cheaper than RedShit.

Other options:

  • InfluxDb (and cloud) – simplicity of ingest
  • ElasticSearch – needs a custom browse component (Kibana too complex)
  • Clickhouse, Vertica, Pilosa and others – “exotic” options, not really worth the effort, esp. given how good TimescaleDb is.

How to ingest:

  • v0: statsD -> influx (fast, cheap, quick, scalable)
  • v2 rsyslog (on each server) => central rsyslog => beanstalkd / kafka => (fan out) postgres (timescaledb) + mongo + influx + es

(Low) code API integrations

19 Wednesday Feb 2020

Posted by in finance, programming, tech

≈ Leave a comment

Tags

What does a distributed / internet operating system look like?

A platform that lets you natively read/write from modern apps? Make apps?

In some sense this is what IFTTT was going for – though consumer focussed, it is IMHO, a very example of a ‘no-code‘ development platform . It’s what Apple’s ‘Shortcuts’ aka Workflow is. It’s what Alexa skills are. I’m not sure Airtable fits, but Airtable got the marketing right.

And now we’re seeing the green shoots of a new generation; pipedream.com and autocode.stdlib.com. These are more developer focussed and provide the “plumbing”, but still need devs to actually make stuff. I’m a lot more excited (perhaps that’s cuz I’m a dev, and these platforms are targeted right at me). I think there’s going to be an explosion of new apps + services, because it’s just so, so easy and quick to knock out an app that uses say Twitter and Amazon S3 … is this how the promise of “serverless” will finally be delivered?

+QQQ

Online IDEs for quick prototypes

09 Tuesday Apr 2019

Posted by in aide-memoire, lifehack, programming, tech

≈ Leave a comment

Tags

#Frontend

Codepen.io: Really great for prototyping layouts and CSS in general.
Recently added Flutter support

  • React: ✔
  • Svelte: x
  • Flutter: ✔
  • JS: ✔ (but not very good …)
  • Private Pens: ✔ (Paid)
  • Secrets: x
  • Deploy to Netlify (or now.sh): x

Codesandbox.io: Webpack + VS Code as an online IDE

#Fullstack (include a VM/Container for development)

Glitch.com: #fullstack, #millenial, #awesome, #secrets

Glitch + Fly.io is very fly!

Repl.it: #fullstack, #secrets, #paid-for-private-repls
Has secrets (.env file that is only accessible by repl owner)

Runkit: The best way to hack nodejs ‘scripts’, IMHO
#backend

Codeanywhere, $8/mo

Stackblitz:
VSCode in the Cloud

#secrets  = .env file that is only accessible by repl owner

Archived

C9.io: Was really nice, but it’s been bought by Amazon; not free anymore, unless you use AWS 😭

Do you know others that should be on this list? Like, subscribe and Leave a comment below!

 

Docker ❤️ Parallels ❤️ OS X (El-Capitan)

11 Wednesday Nov 2015

Posted by in aide-memoire, programming, tech

≈ Leave a comment

Tags

, , ,

Getting started with Docker + Parallels on OS X (using Homebrew)

Update: Sep 2016: This is no longer relevant; best to download Docker for Mac which uses xhyve to virtualize the Docker Engine environment and Linux kernel-specific features on OS X. This allows for much easier sharing of files between your local machine and docker containers.

Want to use Docker on OS X? At the moment, the OS X kernel does not directly support containers like Docker or OpenVZ. So you have to run a VM with docker installed. You then use the Docker CLI to interact with the Docker (daemon) running on the VM. The whole process is actually very easy to setup now that Docker Machine supports the Parallels driver.

Assumptions: You already have Parallels (11, Business or Pro Edition) and Homebrew installed.

$> brew update  # need to brew with a recent version
$> brew upgrade
$> brew install docker docker-machine docker-machine-parallels
… grab some coffee…
$> docker-machine create -d parallels dev1
$> eval `docker-machine env dev1`
$> docker run hello-world

All done!

You are now ready to use Docker, Docker Composer and even Docker Swarm.

#iptables forward an IP address to another

15 Wednesday Apr 2015

Posted by in aide-memoire, programming

≈ Leave a comment

Using #iptables to forward a whole IP address to another (server)

iptables -A PREROUTING -t nat -d $src -j DNAT --to $dst
iptables -A POSTROUTING -t nat -d $dst -j MASQUERADE

Connections from anywhere to $src will get forwarded to $dst (though source IP will be changed to that of eth0 or default outbound device). Useful when moving servers; keeps the old address alive for a bit. Though we lose ‘real’ source address.

For example, if forwarding IP address used by an SMTP server, all email will appear to come from $eth0. If $eth0 (could be the same as $src, but not guaranteed) is privileged, in the sense that it is allowed to relay, then anyone will be able to relay through the SMTP server. But works in a pinch, while DNS changes are propogating through the ‘net.

A somewhat more concrete example. Say you have IP address 1.2.3.4; if you do

iptables -A PREROUTING -t nat -d 1.2.3.4 -j DNAT --to 8.8.8.8
iptables -A POSTROUTING -t nat -d 8.8.8.8 -s 1.2.3.4 -j MASQUERADE

1.2.3.4 is now forwards to Google’s Public DNS Server. You can now use 1.2.3.4 as if it were 8.8.8.8.

Funny programming quotes

13 Tuesday May 2014

Posted by in humor, programming

≈ 1 Comment

Tags

“If Java had true garbage collection, most programs would delete themselves upon execution.”
-Robert Sewell

s/Java/php|perl|whatever/g

 

“C++ : Where friends have access to your private members.”
– Gavin Russell Baker

 

“If debugging is the process of removing bugs, then programming must be the process of putting them in.” – Edsger W. Dijkstra

Proxmox 1.9 – Resize virtual disk

31 Thursday May 2012

Posted by in aide-memoire, programming

≈ Leave a comment

I just had to resize a virtual disk and it was harder than I thought; writing down what I did so it’s easy to find …

  1. shutdown VM 
    qm shutdown 108
  2. resize disk image 
    qemu-img resize vm-108-disk-1.qcow2 +20G
  3. attach pmagic CD; boot to pmagic (using proxmox management web interface)
  4. extend physical partition 
    pvdisplay; lvdisplay; 
fdisk -u=sectors /dev/sda
delete partition 2 (remember starting sector) (‘d’, 2)
create new partition (n) starting at same sector as before; end is defaulted
  5. extend pv 
    pvresize –v /dev/sda2
  6. extend lv 
    lvresize –L +20G VolGroup/lv_root
  7. resize filesystem 
    e2fsck –f /dev/sda2
resize2fs /dev/sda2
  8. detach pmagic cdrom
  9. start VM 
    qm start 108

 

Good instructions for working with Windows (NTFS) disk images is here: http://cauldrondevelopment.com/blog/2009/02/26/resize-qemu-ntfs-image/