Tip jar

If you like CaB and wish to support it, you can use PayPal or KoFi. Thank you, and I hope you continue to enjoy the site - Neil.

Buy Me a Coffee at ko-fi.com

Support CaB

Recent

Welcome to Cook'd and Bomb'd. Please login or sign up.

March 29, 2024, 01:03:24 PM

Login with username, password and session length

Tech stuff you've been learning recently

Started by Z, April 02, 2018, 03:39:04 PM

Previous topic - Next topic

doppelkorn

Quote from: Consignia on August 03, 2018, 01:03:26 PM
Why not set up a virtual environment for python https://realpython.com/python-virtual-environments-a-primer/ ? Should eliminate the need for root command on pip. To be honest I didn't even know pip required root, I thought it would stick everything in user space, but I usually do stuff like that in burnable VMs, so if it all goes tits up, no problem.

I should have mentioned that that was one of the first solutions I came across. I'm even using that primer. But typing in easy_install virtualenv tells you you've not got access to the folder (so back at square one?)

Twed

Try the VM as Consignia mentioned. I do most of my development inside a Vagrant box. https://www.vagrantup.com/

Just be root for anything you feel like in that case, it doesn't matter. Fill the machine with malware? "vagrant destroy; vagrant up" and you're fresh again.

doppelkorn

Sorry if I'm being thick, but I don't understand how to "try the VM" if I can't get packages to install.  Even the thing that gets packages (pip) needs sudo to install.

I feel like everyone knows some crucial piece of information that I'm missing.

Twed

The VM is a computer running inside your computer. Because it's disposable you don't need to worry about what you install on it because who cares?

doppelkorn

No. OK. Sorry.

I know what a VM is, but this is specifically me struggling to do some specific Python thing. Maybe if I go back to what I'm trying to achieve in broader terms that might make some sense.

I wanted to try web scraping with Python, and most tutorials recommend something called BeautifulSoup for that. So to install that according to this tutorial I install pip (using easy_install) then install BeautifulSoup4 using pip. But I ran into the problem of having to use sudo to make easy_install work.

Clever people on the internet say DON'T INSTALL PACKAGES USING SUDO U WIL DEI so I'm like OK what do I do and clever internet people respond USE VIRTUALENV so I'm like OK doot doot hmmmm you need pip to install that so now I'm like:



FYI I'm doing this from the Terminal on a Mac. Is that good? Correct? I have literally no concept of what program to open on my computer into which I can type Python code and make it do stuff.

Twed

If you use a VM and install the packages and do the work on the VM it will not matter. You would be using sudo to install packages on a disposable machine. It would not matter if a rogue package uploaded the entire computer to the dark web because it is just a VM with nothing important on it.

Also just sudo install pip on your main computer anyway, it's fine. It's used by millions of people and is well-vetted. This is one of the circumstances where it's absolutely fine to sudo.

doppelkorn

Quote from: Twed on August 03, 2018, 04:25:05 PM
Also just sudo install pip on your main computer anyway, it's fine. It's used by millions of people and is well-vetted. This is one of the circumstances where it's absolutely fine to sudo.

OK thanks. I did suspect it would be cool. And same for virtualenv i guess?

doppelkorn

Just realised that two posts above my recent dredge, Sebastian Cobb posted a big wodge of Phython he used to scrape CaB. I promise not to steal his ideas tho.

Z

Webpack :(

Then I tried Rollup instead :(

and now I'm back with Webpack :(

Sebastian Cobb

Pyenv/virtualenv makes things a whole lot easier. Also once you've built something, you can run a tool called pipreqs over it to generate a requirements.txt.

Figure it out, and it makes life so much easier.

I figured out docker for work and that was really confusing until it clicked and then, fuck me it's really simple.

I built a bot last weekend to tweet out what's leaving Mubi. That's written in python, it costs me nothing to run it because I built it as a serverless lambda that falls into aws' free tier.

https://github.com/dotarr/MubiNotifier

Sebastian Cobb

Quote from: Twed on August 03, 2018, 01:42:16 PM
Try the VM as Consignia mentioned. I do most of my development inside a Vagrant box. https://www.vagrantup.com/

Just be root for anything you feel like in that case, it doesn't matter. Fill the machine with malware? "vagrant destroy; vagrant up" and you're fresh again.

We use this, it's good, but we are moving to docker containers. They're better I think, as a dev environment you can just map a volume on your host and hot modify your code. It's better. If you use a debian-based image you get apt. I've got some python that transcodes some video and calls ffmpeg externally and it's fine.

Sebastian Cobb

Quote from: doppelkorn on August 03, 2018, 12:11:34 PM
I'm trying to actually do something* in python and I'm falling at the first hurdle.

Trying to install packages using pip or easy_install but from the Terminal on a Mac. This means I have to use sudo, but apparently that is Very Bad because you can't know what's in the packages. So how the fuck am I supposed to install them?

*scrape CaB lol

I'm reading backwards lol. The python implementation on osx is really incomplete. Install python 2.7 or python 3 through homebrew then you'll get a version that isn't protected and doesn't need su to install things through pip (do not simply sudo pip). But yeah, use pyenv/virtualenv.

Steal my hacky python all you like. I accept no liability if it kills someone or is sick on your mum though.

Twed

Quote from: Sebastian Cobb on August 05, 2018, 09:23:32 PM
We use this, it's good, but we are moving to docker containers. They're better I think, as a dev environment you can just map a volume on your host and hot modify your code. It's better.
You can do the same thing in Vagrant though. Not that I am against the Docker idea.

Sebastian Cobb

#73
We do do that in vagrant. And use chef to configure both our local and remote instances (sorry, I should've made that clear really, it didn't occur to me that we did, but we do). There's some diversion in the configuration that happens that I've not seen in our docker implementations, but it could well be due to them being newer and built better from the ground up.

Twed

I think with Docker you tend to get a pretty consistent deploy, which I like.

I don't want my production environments to be derived from dev, because dev usually involves making subtle decisions at the expense of security that can easily get migrated to prod. Would much prefer somebody to create a locked-down, specific release environment.

Sebastian Cobb

I think docker lends itself to that if you do it properly, with the config in the right place.

As it happens I built an aws microservice-based system end-to-end the other week including setting up iam roles and groups properly from the ground up, allowing things only the permissions they needed, and now want to burn everything I've done before it to the ground.

Consignia

I've been messing around with Apache Flink recently. Not really for the technology itself, but as part of my work of building an analytics platform. To get my kata up in the area, I've been developing a Twitter analysis application (using VMs and containers, so nah-nah). It's pretty cool given a few search terms I'm collecting what the general sentiment is around given tags. I've been throwing it against the UK political sphere.



I've been looking at the data over a few weeks, and no matter what time period I run it over, no matter how flawed the sentiment analysis model I use is, no matter how little cleaning of the data, #brexit is always miles away massively negative.

Twed

Very similar to how I've been using observables a lot in everyday programming. You're probably aware of it already Consignia, but if you like the stream-processing nature of that I'm sure http://reactivex.io/ will be up your alley.

Twed

Quote from: doppelkorn on August 04, 2018, 01:39:07 PM
Just realised that two posts above my recent dredge, Sebastian Cobb posted a big wodge of Phython he used to scrape CaB. I promise not to steal his ideas tho.
https://medium.com/@thevatsalsaglani/web-scraping-using-python-and-beautifulsoup-2e54e79415d6 Any use?

Sebastian Cobb

Yeah that's what I did basically.

That beautiful soup makes dealing with xml a piece of piss as well. Recently built a (mostly) microservice based system for work that would pull files from an mrss feed when new ones arrived; maintaining a proper database seemed like overkill for the job (given the metadata would be stored elsewhere) so I just used the key-value nature of s3 to hold that and put it into an sqs queue for a process to download.

If only lambda didn't have a 5 minute limit I might have been able to do the whole thing in that, but you can't rely on being able to shift 10's of gb that fast.

That kibana stuff looks a lot like what we do in sumologic.

doppelkorn

Yeah cheers both - that's exactly what I'm doing. Seeing as the original post by Cobbman has gone, I managed to find the code from a source.

Ta xxx

Sebastian Cobb

Quote from: Sebastian Cobb on May 03, 2018, 12:03:11 AM
I've got one of them microservers (N54L), it's not really doing much more than being a glorified nas/torrent client/vpn.

At one point it was going to be htpc, and it did an alright job with kodi but I was disappointed that it lost v-sync on things like netflix so I fucked it off for a raspberry pi to do kodi + diy ambilight stuff and a firestick.

Part of me wants to sack off the windows server 2008 build that's on it and put either vmware or a minimal arch + virtualbox install on it so I can play about with various os's. I'd have to plug a monitor into it and I can't really be arsed.

Right so I've installed vmware on an SSD on this; I've installed 2 guests on there so far, an encrypted debian 10 + openbox setup for torrent/vpn client and a debian-based nas solution called openmediavault to keep the sharing of files separate (the debian 10 instance and raspberry pi running kodi have it mounted via nfs) I had to buy a reconditioned passport drive to take all my files off my raided 3tb drives and formatted that with ext 4 and LUKS encryption. The files are now copied over.

Now I'm going to format the passport drive, whack LUKS on it and rsync the files back to that. I've got a £10 ethernet/wifi router + nas solution that runs openwrt. The plan is to have that running in another room and have openmediavault rsync the backups to it on a cron job.

It's surprisingly nippy compared to windows server 2008.

Blue Jam

AutoCAD. Had an idea and want to get a design down so I can get someone to machine it for me. Visiting a CAD-CAM workshop to have a look at their laser cutters and 3D printers and I'm getting the RFH about it all. Also trying to get back into Arduino stuff- maybe I could build a few little robots...

Is anyone here a member of a hacklab? There is one near me but it costs £25 per month and I'm not sure I'd be in there all that often.

doppelkorn

Quote from: Blue Jam on September 23, 2018, 02:53:27 PM
Is anyone here a member of a hacklab? There is one near me but it costs £25 per month and I'm not sure I'd be in there all that often.

I was briefly a member of South London Makerspace but then my son unexpectedly arrived and I basically couldn't go. It was cool while I could, tho. I think the fees were £10-£15. £25 seems high.

Z

TypeScript, tbh I'm not sure I see the benefits of it for my project so far... I guess it'd be a lot more useful in work environment or w/e

Sure as fuck is better than Flow though

Sebastian Cobb

Flashed one of these to run openwrt, support luks encryption and rsync so I can use it as an airgapped backup device.

https://www.gearbest.com/wireless-routers/pp_594267.html?wid=1433363

A guy at work reckoned they'd have better throughput than a pi zero, which they probably would without encryption, but if you're slamming files at it you get a load average of about 10.

I've got an edgerouter arriving tomorrow, I intend to make 2 separate networks, one for my computers and one for iot shit like my fire TV and chromecast, with a one-way trust between the two, plus dns routing etc.

Zetetic

Probably horribly patronising but:

1. Are you rsync-ing over SSH? What ciphers?
2. What ciphers and modes for LUKS?

(I'm assuming that thing's processor doesn't have hardware AES but I might well be wrong.)

On 2, it's probably not worth the arsing about but cryptsetup benchmark insists that twofish is slightly less taxing for my machine than AES.

MojoJojo

I ask the possibly dumb question - what do you mean by edge router? For context - I write code for edge routers, but the edge in that context is the edge of an autonomous system, and I doubt you're getting like that. I don't really understand it in a consumer setting.

Google isn't much help as someone has trademarked EdgeRouter.

Sebastian Cobb

Quote from: MojoJojo on September 24, 2018, 09:28:08 PM
I ask the possibly dumb question - what do you mean by edge router? For context - I write code for edge routers, but the edge in that context is the edge of an autonomous system, and I doubt you're getting like that. I don't really understand it in a consumer setting.

Google isn't much help as someone has trademarked EdgeRouter.

Sorry it's a product:
https://www.ubnt.com/edgemax/edgerouter-x-sfp/

I guess it sits somewhere between proper business level gear and consumer grade. You could probably run a small office or something on it.

Twed

I run an Edgerouter X at home. Very happy with it.