upstream updates

en/docs/docker/docker-divergence.md

@@ -0,0 +1,91 @@
+# Divergence
+
+> Last Edit: Fri Oct 1 14:41:40 EEST 2021
+
+Spearhead Cloud's implementation of the Docker Remote API has some added features as well as omissions where the API conflicted with the needs of deploying containers across Spearhead Cloud data centers.
+
+## Features
+
+* [CPU and memory resource allocation](/docker-resource-allocation)
+* Overlay networks
+* Volumes
+* Private registries
+
+# Container behavior and contents
+
+## Extra Processes
+
+### zsched
+In containers you may see a zsched process in addition to your other processes. It owns the kernel threads that do work on behalf of your zone. The PID of this process should always show up as 0, though some versions of ps on Linux filter this out.
+
+### ipmgmtd
+If you don't have docker:noipmgmtd set in your internal_metadata, you will have an additional process ipmgmtd. This is the SmartOS daemon that manages network interfaces and TCP/IP tunables. The default if you use spearheaddocker is that docker:noipmgmtd will be true when you're provisioning a regular LX docker container, and false if you're provisioning a SmartOS container.
+
+### Extra Files
+
+/var/log/sdc-dockerinit.log
+This is the log from the dockerinit process which sets up the container for your initial process and then exec()s it. This log exists only for debugging problems with the way the initial process has been setup.
+
+## Extra Filesystems
+
+### /native/*
+
+If you run mount or df you will see several filesystems mounted in from /native. These are bits from the SmartOS host mounted in to support the LX emulation.
+
+### objfs on /system/object
+This is the SmartOS kernel object filesystem. The contents of the filesystem are dynamic and reflect the current state of the system. See http://illumos.org/man/7fs/objfs for more information.
+
+### ctfs on /system/contract
+This is the SmartOS contract file system which is the interface to the SmartOS contract subsystem.
+
+See http://illumos.org/man/4/contract for more information.
+
+### Exit Statuses
+
+When a container exits the exit status as returned by sdc-docker will currently be different from that which would be returned by Docker Inc's docker. This is due to differences in the way we handle processes within zones. This is currently considered to be a deficiency and should be improved by DOCKER-41.
+
+# Performance of container management functions
+
+Actions performed against spearhead-docker are slower than those same actions performed against a local docker. This is something we are working on, and intend to keep improving over time.
+
+# Docker Remote API methods and Docker CLI commands
+
+In most cases Spearhead has taken great efforts to be bug for bug compatible with Docker Inc's API implementation (see restart policies). Please see documentation for specific methods for any known divergence and file bugs as needed.
+
+Spearhead Docker implements all the API methods necessary to build and deploy Docker containers in the cloud, though there are some missing docker API methods. Here's the list of API methods currently unimplemented:
+
+docker diff, docker events, docker export, docker import, docker load, docker network, docker node, docker pause, docker save, docker service, docker swarm, docker unpause, docker update
+
+## Roadmap:
+
+spearhead-docker development has stopped. While future development will not be happening the system is fully functional and can be used in production. As the list of divergencies from Docker Inc's implementation is getting wider, we recommend setting up Docker hosts directly.
+
+# Images and private registries
+
+Spearhead Docker supports the integration with Docker Hub and third party registries through Docker's Registry v2 API (the deprecated v1 API is not supported).
+
+## Container Logging
+
+Spearhead Docker implements most of the log driver functionality as described in Docker's documentation. The differences in behavior when using these drivers are documented in the Log Drivers feature page.
+
+In addition to the divergence in the use of these drivers, Spearhead Docker handles logging differently than Docker. With Docker, logs are only written to the host when the log driver is set to 'json-file'. With Spearhead Docker the logs are written to the host for every log driver except 'none'. These logs are captured because while the network-based log drivers are good for real-time log analysis, they can potentially lose messages due to conditions like remote host unavailablity and race conditions in startup/shutdown. The host logs are written to local storage by the zoneadmd process that manages the zone and therefore avoids these problems.
+
+The storage for these log files will be within a container's quota but not visible from within the container itself. Operators will see these logs at /zones/:uuid/logs/stdio.log*. These will be rotated to another file in the same directory with a timestamp extension when they reach a max-size value. The max-size can be specified when using the json-file log driver. Otherwise the default (currently defaults to 50M) is set in the Spearhead Cloud configuration. This can be adjusted by an operator through the DEFAULT_MAX_LOG_SIZE property in the docker service's SAPI metadata.
+
+For the user of Spearhead Docker, this additional log when using a log driver other than json-file is not of much use yet, but are important to know about as they do use space from the user's container. In the future, these logs will be automatically consumed and uploaded to Manta which will free this space and give users access to all their logs.
+
+For the operator of Spearhead Docker these additional logs will require some cleanup process for long-lived containers when Manta support is not available.
+
+# Container Labels
+
+Spearhead Cloud uses (and reserves) a number of custom container labels, to provide specific Spearhead features. These are:
+
+The `triton.*` namespace is reserved for Triton specific use cases, these label names are currently defined:
+
+* `triton.cns.disable` (boolean): Can be set on a container to tell the CNS service to not serve records for this instance.
+* `triton.cns.services` (string): Comma-separated list of DNS-name strings for the CNS service.
+* `triton.cns.reverse_ptr` (string): DNS reverse pointer for this container. Used by the CNS service.
+* `triton.network.public` (string): Set on a container, used to specify the external network name the instance will use.
+* The `com.joyent.*` namespace is reserved for Spearhead/Triton specific use cases, these label names are currently defined:
+
+* `com.joyent.package` (string): Set on a container, used to choose a specific package for the container. The value can be a package name like hvm-2cpu-8ram-200disk, a UUID, or the first 8 characters of a UUID (short-UUID).

en/docs/docker/docker-networking.md

@@ -0,0 +1,24 @@
+# Networking
+
+> Last edit: Fri Oct 1 16:09:12 EEST 2021
+
+If fabric networking is enabled for spearhead-docker, each docker container is provisioned with a private nic on the user's default fabric. This allows only other docker containers provisioned on that fabric to connect to each other. The container is able to reach the external internet via Network Address Translation. Each default fabric network is private to a user - one user's containers cannot connect to another's fabric IP addresses, and vice-versa.
+
+To isolate the traffic between different applications or groups of applications, you may want to create additional networks on your fabric with CloudAPI. Any of these user-defined networks can be designated as the 'default' network to be used for provisioning Docker containers. When you change the 'default' network, it affects only the newly provisioned containers. You may specify a different fabric network during provisioning by passing the --network argument in docker run and docker create, e.g.
+```
+   docker run --network=dev-net-123 busybox
+   docker run --network=d8f607e4 -P -d nginx
+```
+The work to support docker network commands is in progress. Follow DOCKER-722, DOCKER-723, DOCKER-724, DOCKER-725 for updates.
+
+All docker containers owned by a user have firewalls enabled by default, and their default policy is to block all incoming traffic and allow all outbound traffic. All docker VMs have a Cloud Firewall rule automatically created that allows them to communicate with each other on all ports.
+
+If you specify the -p or -P options to docker run or docker create, the container will receive an external IP address that is reachable over the public internet, and Cloud Firewall rules are automatically created that allow incoming traffic to the appropriate ports from any IP address. For -P, this means all ports that the VM's image exposes. For -p, this means all ports specified as arguments. Port remapping (eg, -p 80:8080) is not supported. 
+
+If fabric networking is not enabled, all docker containers are provisioned with a nic on the 'external' network by default.
+
+The external network used by a container can be changed by setting the triton.network.public label to the name of the desired external network, e.g.
+
+```docker run --label triton.network.public=external2```
+
+Note that this only overrides the default public network selection. This means that when fabric networks are enabled you will still need to specify one of -p or -P to get the public NIC.

en/docs/docker/docker-remote-api.md

@@ -0,0 +1,58 @@
+# Docker Remote API implementation for Spearhead Cloud
+
+> Last edit: Fri Oct1 14:41:40 EEST 2021
+
+This document explains how to set the Docker client to manage Docker images and containers in Spearhead Cloud's Docker implementation. Familiarity with Docker is assumed. If you are new to Docker, read the [Docker User Guide](https://docs.docker.com/get-started/) to understand some of the basic concepts before you start using Triton.
+
+Spearhead Cloud can be viewed as one large Docker host and does not require you to manage your own Docker hosts. Your Docker client applications can interact with a single API endpoint for a given Spearhead Cloud availbilty zone.
+
+## Connecting to the API
+
+Docker client applications, including the Docker CLI and Docker Compose, can connect to the Spearhead Cloud Docker remote API endpoint to launch and control Docker containers across an entire availbility zone.
+
+Connecting to the API requires an account on Spearhead Cloud, SSH key, and the CloudAPI URL for that data center, as well as the Docker CLI or some other Docker client. Spearhead  provides a helper script to configure a Docker client, including the Docker CLI.
+
+### Docker version
+
+Spearhead Cloud Docker supports clients using Docker Remote API v1.20 to 1.24. For the Docker CLI, this includes Docker v1.8 to 1.12.
+
+You can see the version of your currently installed Docker CLI:
+
+$ docker --version
+Docker version 1.8.1, build d12ea79
+Please install or upgrade the Docker CLI if you do not have it or have an older version.
+
+### API endpoint
+
+Each availabity zone (data center) is a single Docker API endpoint. CloudAPI is used as a helper to configure the client to connect to the Docker Remote API. Determining the correct CloudAPI URL depends on which data center you're connecting to.
+
+### User accounts, authentication, and security
+
+User accounts in Spearhead Cloud require one or more SSH keys. The keys are used to identify and secure SSH access to containers and other resources in Spearhead Cloud.
+
+Spearhead Cloud Docker uses Docker's TLS authentication scheme both to identify the requesting user and secure the API endpoint. The Joyent provided Docker helper script will generates a TLS certificate using your SSH key and write it to a directory in your user account.
+
+### Installation
+You must first install node-spearhead as detailed [here](/spearhead-cli/#install-spearhead-cloud-cli-tool) and a profile.
+
+Once you have the spearhead CLI available you can use the command `spearhead profile docker-setup` to configure your environment for docker.
+
+Once done, you can use either your own Docker CLI or the ones provided by Spearhead *[here](http://127.0.0.1:8000/spearhead-docker-cli/#installation).
+
+*the reason we provide these packaged is to assure compatibility between our Docker Remote API implementation and the Docker Client. There may come a time when Docker decided to no longer support these API's in which case our provided binaries will continue to work.
+
+## Features
+
+Spearhead Cloud Docker offers a number of unique features to our container-native infrastructure, including:
+
+Placement: automatic placement of containers across the entire data center.
+Networking: one or more real IP addresses for each container.
+Resource allocation: specify RAM, CPU, and storage for each container.
+Volumes: container-native volume management.
+Private repositories: image repository management
+Docker CLI commands and Docker Remote API methods
+
+
+## Divergence
+
+The Spearhead Cloud Docker implementation does have some differences from Docker Inc.'s implementation. [See the full list](/docker-divergence).

en/docs/docker/docker-resource-allocation.md

@@ -0,0 +1,50 @@
+# Resource allocation
+
+>Last edit: Fri Oct 1 14:58:23 EEST 2021
+
+When you create a container with spearhead-docker, your container will have an associated "package". The package will be used to determine such things as:
+
+* CPU shares
+* DRAM (memory)
+* Disk quota
+* I/O priority
+
+appropriate for the system your container is provisioned to. The package parameters can be found using the `spearhead package list` command using the [node-spearhead tool](https://github.com/spearheadsys/node-spearhead).
+
+When creating a container with docker create or docker run you can specify the package using the special label `com.joyent.package`. This label can be used with docker create or docker run to choose a specific package. The value should be either a package name like hvm-8cpu-8ram-100disk, a UUID or the first 8 characters of a UUID (short-UUID). For example:
+
+```
+docker run -it --label com.joyent.package=hvm-8cpu-8ram-100disk alpine /bin/sh
+```
+will create a container using the hvm-8cpu-8ram-100disk. If you specify the com.joyent.package label, any -m argument will be ignored.
+
+
+If you don't have a specific package that you want to use but do have a minimum memory requirement, you can specify a -m value for memory and spearhead-docker will select the container package that best matches the resources requested. If there is no package available with the value specified, it will round up to the nearest package.
+
+Regardless of how your docker container was provisioned, you can use the package as a filter for docker ps. To filter you can use the format:
+
+```docker ps --filter "label=com.joyent.package=hvm-8cpu-8ram-100disk"```
+
+which would show you only those docker containers that are using the g4-standard-1G package.
+
+For both lookups with docker ps --filter and container creation with docker create and docker run, you can specify any of package name, package UUID or the first 8 characters of the package UUID. The order of precedence is:
+
+1. package UUID
+
+If the argument is a UUID, we'll only match UUID
+
+2. package name
+
+If the argument is /^[0-9a-f]{8}$/ and matches both a uuid and a name, the package with the name that matches is used. If the argument does not match the short-UUID pattern, and is not a UUID, it's only looked up against package names.
+
+3. short-UUID
+
+If the argument is /^[0-9a-f]{8}$/ and does not match a name, it will be looked up against the first 8 characters of the available package UUIDs.
+
+if none of these match you will get an error.
+
+In order to see the packages for your existing containers you can also do something like:
+
+```docker ps -a --format '{{.ID}} {{.Label "com.joyent.package"}}'```
+
+which will output the id and package name for each container. If there are problems looking up the name of the package because you no longer have access to the package or the package is no longer active, you may see '' as the package name.

en/docs/docker/log-drivers.md

@@ -0,0 +1,11 @@
+# Log Drivers
+
+> Last edit: Fri Oct 1 16:13:20 EEST 2021
+
+With spearhead-docker there is limited support for the --log-driver and --log-opts functions. Important differences include:
+
+* spearhead-docker does not support the 'journald' driver at all
+* the '--log-opt syslog-address' can only be used with the tcp/udp format. The unix://path format is unsupported as that expects to write to arbitrary host locations.
+* the 'syslog-address' option is required when using the syslog log-driver
+* the 'fluentd-address' option is required when using the fluentd log-driver
+* when using any log drivers other than 'json-file' and 'none', additional processes will be running in your container to handle the logging. All hosts specified will be resolved from the container. This allows logging for example on a vxlan network which may not be exposed elsewhere.

en/docs/index.md

@@ -1,7 +1,6 @@
 # Spearhead Cloud Documentation
 
-> Get started with Spearhead Cloud today. Sign-up for a [free trial](https://spearhead.cloud/free-trial).
+> Last edit: Fri Oct 1 14:19:24 EEST 2021
-
 
 ## Overview
 Spearhead Cloud allows you to quickly and easily set-up high performance bare metal containers and virtual machines.
@@ -31,5 +30,5 @@ Spearhead Cloud leverages well known Zones technology to keep tenants from seein
 ### Debugging and Visibility
 Spearhead Cloud provides complete visibility for your systems and applications though dynamic tracing, complex monitoring and native docker logging.
 
-> Last edit: 2019 Mon 27 May 21:41:41 +03
+
 

en/docs/spearhead-cli.md

@@ -1,5 +1,7 @@
 # Spearhead Cloud CLI tool
 
+> Last edit: Fri Oct 1 16:05:33 EEST 2021
+
 The Spearhead Cloud CLI tool allows you to manage your Spearhead Cloud infrastructure. While the Cloud Portal is being developed this tool is the preferred method for managing your infrastructure.
 
 > The Spearhead Cloud Portal will provide all of the functionality available via the spearhead cli tool.
@@ -100,5 +102,3 @@ spearhead instance create \
 ```
 ### instance details
 To view all of your instance's details run ```spearhead instance get -j some-instance```. This will give you a full JSON blob of your instance details. You can optionally pipe the instance details output through the ```json``` command and parse or perform other operations.
-
-> Last edit: 2018 Thu 26 Jul 19:48 GMT+3 

en/mkdocs.yml

@@ -17,7 +17,14 @@ nav:
   - Spearhead Cloud Tools:
     - Cloud Portal: cloud-portal.md
     - Spearhead CLI: spearhead-cli.md
-    - Spearhead Docker CLI: spearhead-docker-cli.md
+    - Spearhead Docker CLI: docker/spearhead-docker-cli.md
+  - Docker:
+    - Docker Remote API implementation for Spearhead Cloud: docker/docker-remote-api.md
+    - Featues: 
+      - Resource allocation: docker/docker-resource-allocation.md
+      - Log Drivers: docker/log-drivers.md
+      - Networking: docker/docker-networking.md
+    - Divergence: docker/docker-divergence.md
 theme:
   name: 'material'
   palette:

ro/docs/index.md

@@ -1,8 +1,5 @@
 # Documentație Spearhead Cloud
 
-> Pornește cu un cont Spearhead Cloud astăzi. Înregistrează-te pentru un [cont gratuit](https://spearhead.cloud/trial).
-
-
 ## Ansamblu
 Spearhead Cloud îți permite cea mai rapidă modalitate de a porni containere și
 mașini virtuale de înaltă performanță.