Create a Task Server

neb · 14 February 2023 12:44

Creating a Task Server

This section will take you through the steps needed in order to create your own Task Manager Server which can then be used to create and work with Tasks.

NOTE: You only need to do this if you need to create a private Task Server and NOT using the public Task Server provided by Yella Umbrella.

You will need an AWS account.

The cloud environment is where the Task Manager will reside and is composed of an EC2 instance and one (or more) S3 buckets.

Creating an S3 Bucket

Go to the S3 Management Console in AWS

Click on create Bucket to navigate to the Bucked Creation Section.

The following section has a large amount of settings that can be adjusted - S3 FAQ - we are going to create a basic S3 Bucket.

General configuration:
Bucket name - Add a name
AWS Region - Select a region - it should be in the same region as your EC2.

The rest of the options can be kept as default for the purpose of this tutorial.

Creating an EC2 instance

Go to the EC2 Management Console in AWS.

Click on instances

This will take you the Instances | EC2 Management Console

Click the Launch Instances button.

The following section has a large amount of settings that can be adjusted - see EC2 Faq for more details which are not covered in this tutorial.

Name and Tags - add a name for your Web Server

Application and OS Images (Amazon Machine Image)

Select Amazon Linux as the AMI
For example, you can select - Amazon Linux 2 Kernel 5.10 AMI 2.0.20230119.1 x86_64 HVM gp2
Make sure that you are choosing 64-bit (x86) Architecture

Instance Type - This is where you will choose the type of machine you will run. It allows you to select from various option which come with different memory/processor configurations.

We suggest using a t3.medium - 4Gbytes and 2 cpu

Key Pair (Login)

You can choose an exiting pair, if you already have one. If not, you can create a new keypair.
NOTE: This setting cannot be changed at a later date.

Click on Create new key pair
This will open the Create key pair window.

Key pair name - choose a name for the keypair.
Key pait type - RSA
Private key file format - Depends on what software you are using for SSH. Choose .ppk if you are going to use PuTTY

Select your key from the drop down. If it does not appear, press on the Refresh button.

Network Settings

Choose a security group - or let Amazon create one (check “Allow HTTPS traffic from the internet”).
We suggest that you lock down SSH to specific IP addresses later.

Configure storage
EBS volume storage

For a Small to Medium Task Servers, 20 GB should be enough.
Choose gp3, it’s faster and cheaper than gp2.
(the linux plus task server install is ~2.5Gbytes. The task server database, if stored locally, does not grow very quickly)
If you intend to do any conversions or other Actions on the EC2, then select enough extra EBS to cover 5 of the largest media you expect.

e.g. if you intend to process films of 50Gbytes each, allocate 20+250Gbytes.
Theoretically, you can expand the EBS volume in the future.

IAM Instance profile
In IAM instance profile, we wish to enable the EC2 to access the S3 bucket created earlier.

Click on Advanced Details → IAM Instance profile → Create new IAM profile

Click on Create Role

Trusted entity type - choose AWS Service
** Use Case** - Choose EC2

Press Next

Add permissions
In the search bar, write AmazonS3FullAccess then press Enter.

Check the box corresponding to AmazonS3FullAccess entry to add the policy name.
Press Next.

Role Details
Add a Role name
Add a Description

Scroll down and then press Create Role. If everything went fine, you should see a Green Banner which says will tell you that Role was Created.

You can now return to the EC2 Console.
In Advanced Details → IAM Instance Profile

Select the IAM from the drop-down list. If it does not appear, press the refresh button then press the drop-down list again.

NOTE: you should investigate limiting the role to the single bucket - this is beyond the scope of this simple explanation.

Termination Protection - scroll down and Enable Termination protection

This will prevent information loss in your EBS volume.

Double check your settings then press Launch Instance

If everything went well, you should see a Success Message.

You will now be able to see the EC2 you created in the AWS Console

Domain Name

If you are happy with the Task Server domain name ending in .taskserver.net, the device can automatically create DNS entries and HTTPS certificates.

For example, you can choose a domain name yourcompany.taskserver.net and configure the task server for this domain, the Server will automatically become available on this domain and as HTTPS.

In cases where you want to use a different root url (different from the .taskserver.net) you can either:
Terminate the HTTPS at an ELB (Elastic Load Balancer), supplying or obtaining your domain name and certificate yourselves.
Place certificate files in a location on the EC2, and specify their locations in the configuration.
The websocket needs to be configured for HTTP. We would recommend no using port 80 or 9000 or something else.

Task Server Device

Now that we have configured the environment where the Task Server will run (EC2 + S3) we can create the Task Server Device.

Go to https://nebula.yella.tv/ and login using your credentials.

In the left side, select Devices then choose the Templates tab. This should bring you to a list of available templates.

From this list choose EC2 Task Server Template and then press the Clone Icon - this will open the Clone Device window.

Device Name - Give the device a name.
Clone Suffix - You add here any text that you want to differentiate between devices if you have more than one with the same name.
Description - Add a description for this device.
Location - Select a location. Should be in the same area where your EC2 and S3 are situated.
Device Type - Amazon Linux 2 service (it will default to this)
Create Template - No
Clone Type - Device and all project and processes

Press on Clone device to start the cloning process. Once it has finished, the device will appear in your Devices list. with Status of

Click on the cog wheels icon to expand the device.

Click on the this icon to expand and see the Processes on this Device.

Task Server Configuration

Configuration of a Task Server devices requires configuring several processes.
Prerequisites:
Your domain name - we will use a .taskserver.net domain in this tutorial
Your S3 bucket link - it should look like s3://bucket-with-the-name-you-created

Configure all processes for this URL, pre-pended with https:// (or http:// if you want to run without SSL - not recommended).

Generic API V2 Process
Websocket Server - Add the websocked secure link

Websocket Authentication - copy the string here, we will be using in another process

Websocket Distribution Process V4 - press the pencil icon

Add the server domain in the following box.

In the API section - paste the Websocket Authetication that copied in the previous step in the API Key String section

2nd factor process
2nd Factor URL - Insert your domain name with the https://

Action Manager
Server URL - Add your domain URL in the Server URL section

Task List Server V1

Server Name - Add the Server name
Server URL - Your domain name

Storage configuration - Amazon S3
Storage Type - Amazon S3
S3 Root Path - The link to the S3
Access Key ID - Your S3 access key id
Secret Access Key - Your S3 secret access key

PostgreSQL - Task Manager- Optional
By default the Task List Sever V1 process is using a SQLite database.
You can configure it to use a ProsgreSQL database…

JSON File Path & .json file name can be changed, just make sure they match.

 taskmanagerdb.json
 {
    "type": "postgres",
    "connection": {
          "host": "your host",
          "port": "your port",
          "user": "username",
          "database": "taskserverdatabasename",
          "password": "yourpassword",
          "ssl": {"ca": "/yourfile.pem", "rejectUnauthorized": true}
   },
 
   "pool": {
     "min": 1,
     "max": 100
   }
 }

If you do not want to use SSL, you can leave the CA field empty set rejectUnauthorized to false.
If you are using a self-signed cert, add it in the CA field and set rejectUnauthorized to true.

Creating the Device

Once you have finished configuring the processes, you can press on the download icon
coresponding to your device.

Confirm on the next message.

Press download on the next message.

A zip file containing the required files will be downloaded.

Connecting to EC2

You can use Putty or Kitty to connect to your EC2 server.

From the EC2 console, you will need:
Your - Public IPv4 DNS
The Key - The key you saved when creating the EC2. It’s a .ppk file that you downloaded when creating your EC2 instance.

Putty

Go to Sessions - put the Public IPv4 DNS in the Host Name (or IP address) field.

Go to SSH → Auth → Credentials
Private key file for authentication - Browse for you .ppk file.

Go to Sessions → Saved Session → Enter a name → Save
This will save the session configuration so could access it in the future without having to re-input the details.

Click on Open.

When prompted with login as: ec2-user and then press Enter
If everything went fine you will see a message that says:
Authenticating with the public key “your key name here”
Last login:

   __|  __|_  )
   _|  (     /   Amazon Linux 2 AMI
  ___|\___|___|

And then you should have terminal access to your EC2.

Copying Task Server Files

First we will need to connect to the server using another piece of software which will allow us to upload the required files. WinSCP is a good choice for this.

WinSCP

Host name - Public IPv4 DNS
User name - ec2-user

Click on Advanced → Authentication

Browse for your .ppk file then press OK.

Once you have entered all the details, press save to save the session details.
Site name - give the session a name then press OK.

It should appear in the right side column.

Select your session and then press

If all went well, WinSCP should contain two windows. The left one allows you to browse content on your device, while the right one allows you to browse content on your EC2 Server.

Find the device ZIP that we previously downloaded and unzip its contents. You should see 4 files similar to these:

We need to transfer these files to our EC2 in /user/ec2-user/

In WinSCP, make sure that you are in /user/ec2-user/

Select the the 4 files we have previously unzipped and drag them in right window of WinSCP

And upload window will now appear

Once finished, it should look similar to this:

You can now close WinSCP

Installing the Task Server

Go back to Putty and connect to your EC2.

To check that the required files are present, type ls and then press enter.
This will list all the existing files in the ec2-user folder.

[youruserhere]$ ls
install-40702.sh  Neb_cristian_stelaris_40702  readme40702.txt  service.pattern

Go back on your Desktop and find the readme.txt file in the folder that we unzipped and then open it in notepad.
This readme file contains all the commands that we have to run to start the device, and also how to do several other operations.
Your install command should look similar to this.

chmod 755 install-40702.sh; ./install-40702.sh

Copy and paste this command in the terminal and then press Enter.

If all went well you should see a message prompting you to:

To run it now use:
- sudo systemctl start YourDeviceName.service
- or reboot

Copy the first command
sudo systemctl stop YourDeviceName.service and then press Enter

If all went well, in you Nebula Dashboard you should be able to see all processes Green Running.

List of Commands - The exact commands for your device can be found in the Readme.txt file.
start the device:
sudo systemctl start yourdevicename.service

stop
sudo systemctl stop yourdevicename.service

restart
sudo systemctl restart yourdevicename.service

disable startup at boot
sudo systemctl disable yourdevicename.service

enable startup at boot
sudo systemctl enable yourdevicename.service

view logs
journalctl -u yourdevicename.service

view last logs
journalctl -e -u yourdevicename.service

follow latest logs continually
journalctl -f -u yourdevicename.service