What's new

Archived LRP - Labs Infra

PIVX Labs

Administrator
Staff member
Code:
Name: LRP - Labs Infra
Term: 3 Cycles
Cycle Amnt: 2,500
Total Amnt: 2,500
Author: JSKitty
Receiver: PIVX Labs
Address: DLabsktzGMnsK5K9uRTMCF6NoYNY6ET4Bb
Created: 26-11-2023
Status: Active
Vote Hash: d4082035a77319d516514a5b757cca479115751d6175a40cefea39a981d247f0



Proposal Abstract
In the last month, Labs has encountered a dozen infrastructure-related stressors, primarily being:
- DDoS Attacks: these revealed a need to bunker-down our servers, proxi-fy our sensitive services like PIVCards and PIVi.
- Rogue Providers: likely due to the above attacks, and investing heavier in additional servers - our old providers, quite literally, suspended and confiscated all PIVX Labs servers without warning, demanding KYC with a 24h termination of account.

Since the release of PIVCards, I spent some time polishing it myself while the Labs Team focused on MPW - then came the attacks, eating an entire week of time as we bunkered down a multi-server solution - to top that off, our provider decided to stranglehold our servers! This was a headache that has eaten another full week of my work and personal time, and caused me some personal burnout that I am fighting through, from the lack of breaks - fortunately, with some old-but-good-enough backups, we have not lost any major data, as it currently seems - nevertheless, with a trustworthy provider, with strong in-house infrastructure, human support agents, and a give-no-fucks attitude to the content hosted on their service... Labs would have faught off this last week much easier.

That is why we are now moving to Njalla, "Considered the worlds most notorious "Privacy as a Service" provider", Njalla has the same founder as The Pirate Bay and countless other infamous anti-censorship services, including supporting WikiLeaks, Njalla have the same attitude as Labs, and we have a direct PGP-encrypted chatline to them - why wouldn't we?




Njalla (Onion) - Njalla (Clearweb)
Njalla's hardware is based in Sweden, and unlike most; they are not a reseller, but direct hardware-as-a-service.

/ˈɲalla/ (Sami),
Small hut in the Sápmi forest, built to protect against predators.
1701016039897.png



Proposal Plans

We are in the process of testing Njalla and rolling out Labs services using them - PIVi and our other micro-services have been moved over, the Labs Cold Pool is also being moved over as we speak, during this month's testing phase, we'll cover the servers from Labs' existing fund (contributed to by our Quality Control and other in-house LMPs), and this gives the DAO a month to vet out the service, and gives us time to chat further with them about custom orders, domains, and other useful services Labs may need in the near future.

This proposal aims to cover all Infrastructure costs, including but not limited to:
- All current and future Labs domains (MyPIVXWallet.org, potentially PIVCards, and room for additional if needed).
- MyPIVXWallet Primary server. (Hardened MPW-dedicated server).
- MyPIVXWallet Backup server. (Hardened MPW-dedicated server, used for load-balancing and/or test deployments).
- Labs Cold Server. (Used for Labs micro-services such as Labs Cold Pool, PIVi, Prodder).

- PIVCards Master Server. (Runs full nodes of PIVX Core, Bitcoin Core, as well as the PIVCards Codebase, securely holding up-to 0.1 BTC of funds for operation).
- PIVCards Proxy Micro-Servers. (NOT hosted under Njalla due to Proxies needing to be located internationally, these servers allow PIVCards to operate in select global regions, like UK, US, EU, Canada, etc).


The costs are estimated to be around 500 to 800 Euros a month depending on the final hardware we acquire, converted to PIV at current rates, again, as a 'maximum band' number, meaning we could theoretically run lower, but it leaves little headroom for upgrading or changes of plans under a month (superblock cycle).

The proposal can be updated with server specs if desired - but currently we only have estimates, while we test the comfortable server capacity Labs services can operate safely under - our last DDoS attacks proved that running "just good enough" servers is NOT a good idea, so we need to start thinking with operational headroom.

Njalla is indeed a pricier provider, however, if it means PIVX Labs infrastructure remains in trustworthy hands, that will not confiscate our servers at random, will assist us under DDoS attacks, and communicates directly with us, the price remains worthwhile, as it will save us weeks of time, that could be better spent (and ironically, would lead to being more expensive - as myself and the team are forced to handle these issues instead of working on PIVX, wasting our time and thus wasting our DAO funds)
.
 
Last edited:
Sorry you and the rest of Labs had to go through all this.
But, glad the infrastructure will come out WAY stronger and more resilient than before.
Well written proposal and it makes sense.
Cost is definitely worth it, given how valuable MPW and PIVcards etc are to PIVX.
Thanks for all you do!
 
Hi, I'm curious, what are the server/vps requirements? .. apart from ability to checkout using $pivx?
 
Hi, I'm curious, what are the server/vps requirements? .. apart from ability to checkout using $pivx?
He literally said that they are a guess at this point.

"The proposal can be updated with server specs if desired - but currently we only have estimates, while we test the comfortable server capacity Labs services can operate safely under - our last DDoS attacks proved that running "just good enough" servers is NOT a good idea, so we need to start thinking with operational headroom."
 
Would it be possible to filter the traffic through Cloudflare? .. just questioning.. it seems setting up a dedicated server as a DDoS protection layer would be overkill.. thinking loud..

what was the previous host that you guys had issues with? .. is there a chance it was hetzner? .. many use hetzner despite knowing that they are against any usage of their servers for crypto needs.

also, isn't mypivxwallet hosted on github?

curious.. why pivx needs to pay for bitcoin core hosting?

--

and, where can we read more about pivcards, please? .. lots of questions :D :D
 
Would it be possible to filter the traffic through Cloudflare?
We already do (always have), it did not stop our old servers from a complete lockup - we'll now have server-level protection directly from Njalla at request, plus beefier servers, making this much easier to deal with in the future.

what was the previous host that you guys had issues with? .. is there a chance it was hetzner?
A mix between Cloudzy for larger servers (prev. called Routerhosting, which Labs has been using since 2021, so we trusted them a lot), and BlueVPS for smaller international servers.

also, isn't mypivxwallet hosted on github?
The source code is on GitHub, and we do have a 'Bleeding-Edge' instance on GitHub Pages (https://pivx-labs.github.io/MyPIVXWallet/), but it is not considered stable and is only periodically updated - GitHub's hosting isn't intended as a dedicated service, more useful for running tests, backups, etc - GitHub Pages cannot be optimised for SEO, nor do we have control over other files that MPW needs for Search Engine and Crawler Bot purposes.

curious.. why pivx needs to pay for bitcoin core hosting?
It's necessary for PIVCards to operate: http://mypivxwallet.org/pivcards
 
We already do (always have), it did not stop our old servers from a complete lockup - we'll now have server-level protection directly from Njalla at request, plus beefier servers, making this much easier to deal with in the future.


A mix between Cloudzy for larger servers (prev. called Routerhosting, which Labs has been using since 2021, so we trusted them a lot), and BlueVPS for smaller international servers.


The source code is on GitHub, and we do have a 'Bleeding-Edge' instance on GitHub Pages (https://pivx-labs.github.io/MyPIVXWallet/), but it is not considered stable and is only periodically updated - GitHub's hosting isn't intended as a dedicated service, more useful for running tests, backups, etc - GitHub Pages cannot be optimised for SEO, nor do we have control over other files that MPW needs for Search Engine and Crawler Bot purposes.


It's necessary for PIVCards to operate: http://mypivxwallet.org/pivcards

Thank you. All the best!!!
 
Top