From 4e8368f4d847e5c1352302fc53658dfab2c72a9b Mon Sep 17 00:00:00 2001 From: Phil Burton Date: Fri, 22 Feb 2019 15:32:18 +0000 Subject: second and third talk --- day1/first-talk.txt | 97 ++++++++++++++++++++++++++++++++- day1/no-sql.txt | 52 ++++++++++++++++++ day2/gitlab.txt | 120 +++++++++++++++++++++++++++++++++++++++++ day2/rest-apis.txt | 142 +++++++++++++++++++++++++++++++++++++++++++++++++ day2/second-crypto.txt | 51 ++++++++++++++++++ 5 files changed, 460 insertions(+), 2 deletions(-) create mode 100644 day2/gitlab.txt create mode 100644 day2/rest-apis.txt create mode 100644 day2/second-crypto.txt diff --git a/day1/first-talk.txt b/day1/first-talk.txt index ffd2f65..da8689e 100644 --- a/day1/first-talk.txt +++ b/day1/first-talk.txt @@ -1,8 +1,101 @@ -micro services - perf +# micro services - perf -glu mobile - 1st speakers company (microservices) +## Overivew + +glu mobile - microservices + +- Use docker +- HTTP processing +- Amazon ECS +- NOSQL Redis + couch base +- 40M votes a day +- 1M active users +- 14K req/m + +## Microservices + +- Small +- Simole +- easy to deploy +- upgradable + +Use Base images +Name by version and never change (Freeze them) + +## background processing +- Execute externally +- Shutdown functions +- Job server +- fastcgi_finish_request + +## HTTP Compression + +- gzip on +- gzip_types +- gzip_proxied any +- gzip_comp_level 5 +- gzip_min_length (default 20) +- Smaller than 1500B means it will always fit in one TCP packet + +If content_length is not set then nginx will compress always + +Laravel/Lume/Others add middleware to add content_length + +## Caching +- HTTP2 Server push +- Repeated server calls HTTP1 +- 304s + - Last-modified + - cache-control + +## No SQL +- Mongo/Redis +- Fight network I/O +- save hdd space +- compress large messages +- serialise better +- message templates +- add ttl expiry +- shorten cache keys + +## Fast caching +- PHP Redis +- Predis - high memory, slower +- Persistent connections available + +## Deployment EC8 +- 0 Downtime +- quick rollout +- ELB health-check +- Auto scaling + +## Hardware upgrades +- Doubling the hardware, halving the # of machines + +## VPC +- Closer together +- Same location +- utilise internal networks +- other thoughts + - Async I/O + - swoole + - HTTP/2 + - Tars + - binary (80% reduction) + +## Managing images +- Don't use aptitude (apt) +- Build from source +- fpm and Nginx together + - Separate not helpful + - No need for TCP/IP overhead +- supervisorD - everywhere + +# Useful terms +aerospike couch base php_swoole +supervisorD new relic bugsnag amazon cloud watch diff --git a/day1/no-sql.txt b/day1/no-sql.txt index 5ce9723..ccb6a4a 100644 --- a/day1/no-sql.txt +++ b/day1/no-sql.txt @@ -1,3 +1,55 @@ +# Non-Scalar Data + +## Redis sets SSAD + +- key binding - safe string +- sets of data +- taggable + +## Document data stores +- Usually JSON +- Richer +- MongoDB - single master +- CouchDB - multi master +- ES - text search index + +## MongoDB +- pecl/mongodb +- mongodb/mongodb + +docs are PHP objects or assoc arrays + +## Relational Database +- normalisation + +- postgres + - HSTORE + - JSON/JSONB + + +# Querying data + +Sets - SISMEMBER and SMEMBER +- Aggregations +- buckets +- postgres - non standard + +## Redis +- Can add to sets +- Atomic +- fast +- counters + +DONT: RETRIVE MANIPULATE STORE + + Do we do this at d3R? + + redis bad at scaling + + joned.in/25887 + + +# Useful links https://redis.io/commands/sadd https://www.postgresql.org/docs/9.1/hstore.html https://wiki.openstreetmap.org/wiki/PostgreSQL diff --git a/day2/gitlab.txt b/day2/gitlab.txt new file mode 100644 index 0000000..2ce3906 --- /dev/null +++ b/day2/gitlab.txt @@ -0,0 +1,120 @@ +# Dev to prod with GitLab CI + +## Overview +- Stefan +- bitExpert AG (Germany) + +## Gitlab +Old wikipedia description: + Web based git repo manager with wiki and issue tracking +## Beyond CI/CD + - Dev, Ops, CD/CO + +## Installation +Lots of ways. +Docker. +- Use a registry +- Sonatype Nexus +- Nexus installation for repo manager +- Reverse proxy for routing, and running multiple instances + +### Traefik +Simple to configure +Configure +Can listen to socket, and other things +Can do lets encrypt stuff + + +## Crating projects +Magento as an example +Slightly more difficult to install +Has some complication + +1. Composer install +2. Add to git and push +3. Need a git lab runner (Can host the runners or use SASS offering) + (docker container for runner) + +(Gitlab instance and runner installed via docker) + +Access runner overview page and grab registration token +Create runner instance +gitlab-runner gitlab-runner register + - token + - description + - tags + - run untagged builds? + - current project? + - executor? (docker, shell,.. etc) (more control if we use docker) + - default docker image +-- This creates the runner config file +-- Edit and amend, e.g. add volumes of shared composer stuff ***Should we do this at d3r?*** + +## Managing Secrets +add to project, can use in groups + +## Add .gitlab-ci.yaml (similar to travis) + -- define Image + -- define job scripts + +## http-basic auth via composer +What is this? + + +## Gitlab services +Multiple images should spin up first, then jobs can run +Health checks built in, cheks exposed ports +Spins up second container waiting for ports to be available then spins down +(Gitlab says you cant 100% trust this to coinfirm all is working) + +### add mysql service +1. Define image, define variables (passowrd) +2. define things it needs + +## Build piplines +- test +- build +- build staging + - define where in "data center" the staging stuff is + - Define git registration + - dine staging url +- Build creates tar file ready for deploy +- deploy! +- when manual trigger - Dont auto build and deploy everything to staging and production, Allow us to test te staging +env first and then manually kick off the deployment for production + + +## Problem with this set-up +- Multi developers means we're not sure of staging set-up +- Staging per developer? +- What about a staging per feature branch? + - yes + - Review Apps + - documentation not great + - Dynamically create staging environments + - make sure on_stop + - only run for merge_requests (fairly new, used to have to do "all branches but not master") + - define stop_review to pull down containers + +1. Create merge request +2. Test and build begins +3. It then gives you the urls for testing +4. Merge. Then staging is deployed +5. Then manually hit up production + + + +## Questions +Create a stadard web app +Can Import from github!!! +Autoi dev ops? + - Auto turned on + - Not sure how good it is + - maybe we just disable +Can we lock down how things get deployed + - ROle based auth is bad + - Dont have fine grain control (most of the team need nainainer role) + +# Useful links +https://www.sonatype.com/ +https://traefik.io/ diff --git a/day2/rest-apis.txt b/day2/rest-apis.txt new file mode 100644 index 0000000..a087a52 --- /dev/null +++ b/day2/rest-apis.txt @@ -0,0 +1,142 @@ +# First class rest APIs with symfony + +## Who +- @michaelcullumuk +- Works for bud. (2 weeks ago) +- core team at syfony +- fig working group +- fig security + +## WTF IS REST? + +- Paper by roy fielding +- dry and boring +- Representational State Transfer (REST) +- Communicating state + +### How to comply to REST + +- Uniform interface + - Each api works the same at a ui level + - Gives all data to client that client can then use to give back and modify + - use mime types + - Hyperdata. Link to other data. + +- Stateless + - No session + - Self contained in one request + - Auth in the request + +- Cacheable + - If not cacheable, you need to tell clients this + +- Client-Server need to be distinguishable + - Independently build client and server + +- Layered system + - May not be talking to an end user + - May be other applications or proxies or API in between + - (like micro services talk adding auth or similar at certain APIs) + +- Code on demand + - Be able to run code on the server + - Legacy. + - bit weird + - considered optional + +- USE HTTP WELL + - HTTP is your friend + - RESTful does not discuss http + - HTTP VERBS + - USE THEM ALL + - (Google crawling with GET, where they hit a GET verm delete end point) + - USE RESPONSE CODES + - Rate Limiting + use codes (429) + - 503 for when there's issues + +- In symfony + - Use the constants + +## Errors +Exception controller +Does logging +Return our json response +getErrors() handles error once in one function, included for all controllers + +## DTO ALL THE THINGS +- in PHP we tend to use arrays lots +- Decode some JSON and might put it into array +- bad as we have no typing +- We can use typing to move complication and validation of data to PHP level +- Reduces typos etc +- In symfony we can auto-hydrate stuff + - mapAndPersist + - A dataTransferObjectConvertor + +## Validation +- Exceptions + - Exceptional case + - Should expect edge cases and failings + - Remove control from controllers + - isValid in the controller forces us to have logic in the controller + - instead try and persist an object, throw new exception + +## Output + - Symfony serializer component + - Choose output types easily + - Transformers to migrate the data from an entity and map to response data + - Allows us to focus on API respires not database layer entities + - PHPLeaue FRACTAL (for transformers) + - Transformers allow us to be explicit! + - Using fractal can build our json but bhandke collections, pagination etc + +## Nobody likes pagination + - Pain in the arse + - Clients need to work out logic for reading + - Server side we have extra logic, limits etc + + How to do it then? + - Do it + - Link to next and previous (hyper media) + - Detail current page + - Detail total number of pages + - Assuming you know + - Disabling totals speeds things up + - Perhaps client option + - Use paginator tools + - Really simple controllers + +## Sorts and filters +- query string + - pagiantion + - sorting + - includes + - filters + +## Tools +- Api platform + - EZ PZ +- FOS REST + - automatic routing + - slugs etc + +# RECAP +PRINCIPLES OF REST +USE HTTP WELL - verbs and codes +ERROR handling, abstracting validation. Handle expections for response codes +DTOs and param convertor +Validation bubbles, catch and re-throw +Fromatting output wiht transformes +Pagination - annoying but easy, if we p[ass via repos. +Sorts and filters similar to Pagination +Tools!!! Use em.] + + + +# Useful Link +https://github.com/FriendsOfSymfony/FOSRestBundle +https://github.com/whiteoctober/Pagerfanta +https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm +https://en.wikipedia.org/wiki/Data_transfer_object +https://fractal.thephpleague.com/ +https://api-platform.com/ diff --git a/day2/second-crypto.txt b/day2/second-crypto.txt new file mode 100644 index 0000000..18d5324 --- /dev/null +++ b/day2/second-crypto.txt @@ -0,0 +1,51 @@ +# Crypto + +## Overview: +Why do we need crypto? + +- Confidentiality +- Key Ex +- Identity +- Authentication + +## Asymetric Cryto +- Symmetric crypto faster, asymmetric slower and computational expensive +- RSA + - slightly old + - Needs 2048+ size keys + +## eliptic curve crypto +- Dot function +- Key exchange with shared key created from a -> aG -> aGb <- bG <- b +- Week to man in the middle + +## Web of trust +- How can we safely pass the key over an unsecured channel +- Just blindly trust on first use (like ssh) + +## The trusted third party +- Get a 3rd actor to meet Alice and Bob to add trust + +## hash functions +- md5 weak +- sha good + +# Merkel tree +- Hash chunks +- Combine chunks and hash +- Can work out which parts are wrong +- good for static data +- hash tree + +## BLock chain +- Use block chain to share trusted messages + +@giveupalready +https://github.com/carnage + + + + + +-------- +Web of trust, trust once, trust forever -- cgit v1.2.3