Re-publish Kafka record not setting partition number?

Well, I plan to re-publish Kafka records that my app is consuming to another Kafka topic on another host (internal only). This sort of lets other groups consume data from this topic without having to create connections outside of the private network, since the original data source is a 3rd party vendor. They can then also do their own transformations or other business logic on it as if they were consuming from the original source. Delay from source to re-publish should be minimal. Will still be close to real time.

What I normally do when producing records is not to set a partition by default. Would this be okay when re-publishing records? Will there be conflict with original record partition number to the target topic partition?

ANSWER

How many partitions does your target topic have? Do you have freedom to create as many partitions as the source?

(1) If you can create 1:1 partitions then my suggestion is to retain the original partition number of that record when you publish to the destination topic.

(2) You can also assign the partition in a round-robin way when you have lesser partitions on your target topic. Save metadata info of that record in the headers, such as original partition assignment and etc. Perhaps that will be useful down the line for your consumers.

(3) Lastly, you can NOT assign a partition number. Instead, let Kafka do the magic for you. When no partitions are set, Kafka will determine the next partition in a round robin method. You can do this with a null value instead of an int for partition number.

Bottom line is that it depends on the project requirements, I suppose, or more so on the infrastructure of your destination Kafka.

Fail2ban apache-noscript jail violation rules?

Which actions specifically are considered as violating apache noscript fail2ban module? I see that it correctly banned the ip that was looking for strange urls like “/admin” “/login” etc, but also blocked my ip and I checked the log for my ip and I didn’t see any suspicious actions. So in one case it worked good in another it didn’t.

Btw one of this line was considered as apache noscript violation. Why?

myip - - [05/Sep/2020:12:55:52 +0000] "GET /css/theme/jquery-ui.custom.css HTTP/1.1" 
myip - - [05/Sep/2020:12:55:52 +0000] "GET /img/icons/manifest.json HTTP/1.1" 
myip - - [05/Sep/2020:12:55:52 +0000] "GET /css/some.css HTTP/1.1" 
myip - - [05/Sep/2020:12:55:52 +0000] "GET /css/bootstrap.min.css HTTP/1.1" 
myip - - [05/Sep/2020:12:55:52 +0000] "GET /ww.js.map HTTP/1.1" 
myip - - [05/Sep/2020:12:55:52 +0000] "GET /ww.js.map HTTP/1.1" 

Btw, why are not attached the log files in the mail report? I edited in the fail2ban configuration the log to /access.log from /*access.log so hopefully it will help with this.

Go to Source
Author: luky

Ubuntu Installation Stuck at “update-grub”

I have gone through all the similar questions asked about this problem and tried the workarounds mentioned over there but nothing works. Like the ” ps | grep ‘dmsetup create’ ” workaround simply doesn’t work, because I don’t get any output displaying any PID (so I can’t kill any process), then the one regarding deleting 30_os-prober just deletes the file but has no effect on the installation. I am trying to do a clean install on a HDD, can anyone please help me solve this problem. [I am installing Ubuntu LTS 20.04.1 from a live USB]

Go to Source
Author: Utkarsh Jha

cant restore nuget packege on ubuntu after upgrae to .net core 3.1

i am running .net core 2.1 on Ubuntu 18.04 so far so good

i upgrade my project to .net core 3.1 and installed the 3.1 SDK

since than i cant use ‘dotnet restore’

this is the full output

/usr/share/dotnet/sdk/3.1.401/NuGet.targets(128,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/repos/myproj/portal/PortalApi/PortalApi.csproj]
/usr/share/dotnet/sdk/3.1.401/NuGet.targets(128,5): error :   The SSL connection could not be established, see inner exception. [/repos/myproj/portal/PortalApi/PortalApi.csproj]
/usr/share/dotnet/sdk/3.1.401/NuGet.targets(128,5): error :   The remote certificate is invalid according to the validation procedure. [/repos/myproj/portal/PortalApi/PortalApi.csproj]

any idea why ?

thanks

Go to Source
Author: David Munsa

Не загружается проект на Django с VPS Ubuntu, после настроек gunicorn, supervisor и nginx

Всем привет! Столкнулся с проблемой, произвел все настройки на сервере, проверил работу gunicorn, запустил проект напрямую командой gunicorn Superman.wsgi:application --bind 0.0.0.0:8001, сайт запустился, но без static файлов. Далее установил supervisor, так же произвел отдельную настройку под него и настроил nginx, при проверке supervisor sudo supervisorctl status Superman сервер ответил, что он работает, при проверке nginx sudo nginx -t сервер так же отвечает положительно successful, но когда я забиваю IP своего сайта в строку поиска браузера сайт не прогружается, ошибка – Сайт 91.228.152.32 не позволяет установить соединение, nginx пусто. В чем может быть проблема, возможно я не правильно настраиваю proxy_pass? Настройки прилагаю ниже, ПРОШУ ПОМОЧЬ В РЕШЕНИИ ПРОБЛЕМЫ, сутки ищу решение и ничего((

Настройка Gunicorn

NAME="Superman-test"                                  
DJANGODIR=/webapps/Superman-test/Superman             
SOCKFILE=/webapps/Superman-test/run/gunicorn.sock 
GROUP=www-data                                      
NUM_WORKERS=3                                     
DJANGO_SETTINGS_MODULE=Superman.settings             
DJANGO_WSGI_MODULE=Superman.wsgi                     

echo "Starting $NAME as `whoami`"

cd $DJANGODIR
source ../bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH


RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

exec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application 
  --name $NAME 
  --workers $NUM_WORKERS 
  --user=$USER --group=$GROUP      
  --bind=unix:$SOCKFILE 
  --log-level=debug 
  --log-file=-

Настройка Supervisor

[program:Superman]
command = sh /webapps/Superman-test/bin/gunicorn_start 
user = root 
stdout_logfile = /webapps/Superman-test/logs/gunicorn_supervisor.log 
redirect_stderr = true 
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8 

Настройка Nginx

upstream Superman-test_server {
  server unix:/webapps/Superman-test/run/gunicorn.sock fail_timeout=0;
}

server {

    listen   80;
    server_name 91.228.152.32;

    client_max_body_size 4G;

    access_log /webapps/Superman-test/logs/nginx-access.log;
    error_log /webapps/Superman-test/logs/nginx-error.log;
 
    location /static/ {
        alias /webapps/Superman-test/Superman/static/;
    }
    
    location /media/ {
        alias /webapps/Superman-test/Superman/static/app/image/;
    }

    location / {
        if(!-f $request_filename) {
            proxy_pass http://Superman-test_server;
            break;
        }
    }

    # Error pages
    error_page 500 502 503 504 /500.html;
    location = / 500.html {
        root /webapps/Superman-test/static/;
    }
}

Go to Source
Author: Jony

Combining Two Containers

I’m working on a Windows laptop and running Docker for Windows on it. I want to run an Ubuntu container with a specific version and a MySQL server on it. So the question is, do I have to download MySQL on the Ubuntu container or can I run 2 containers (Ubuntu and MySQL) and combine them? How do I combine these 2 containers?

Go to Source
Author: user19215

Blocked from using pip3

Default python is 2.7; default python3 is 3.6, which is the version I use; Ubuntu version 18.04.
I’m trying to get newly downloaded python libraries to be installed in a python 3 library.

    charles@BW2:~$ pip3 --version
    Traceback (most recent call last):
      File "/usr/bin/pip3", line 9, in <module>
      from pip import main
    ImportError: cannot import name 'main'

What to do?

Go to Source
Author: cbro630

How to show Snap app icon properly on Ubuntu Unity desktop

Had this issue where after I logged into my Ubuntu desktop all Snap apps icons do not show properly. Instead I see a plain icon in place of what the app icon should be. Have you the same experience?

This might have been fixed in later Ubuntu LTS releases, I really don’t know. But I am still on 16.04 Xenial Xerus until its EOL. Also, the reason is that I love the Unity Desktop. The Gnome Shell replacement just doesn’t cut it for me.

I have been looking for a solution on this issue for some time. The 2 main things people point out are:

– Add /snap/bin to the /etc/environment file, and;

– Something to do with the profile.

Neither of those solutions worked for me. But it did work for others according to what I have read so far.

ANSWER

To fix this icon issue, create a symlink of the Snap’s .desktop shortcut file to your user’s .local/share/applications directory.

It is as simple as that. Once done the problem will go away, and you will be left with a nice-looking icon that came with the Snap installation. This won’t go away on following restarts.

For more details on how to do this fix exactly, refer to this article – https://www.joseyamut.xyz/2020/08/13/getting-that-ubuntu-snap-icon-right-on-unity-shell/

How to stop truncate command safely

I am trying to empty my syslog.1 file which was flooded with some messages and has the size of 77 GB. I did

sudo truncate -s 0 /var/log/syslog.1

but the command is taking more than 2 hours to return. Is it safe to stop it by Ctrl-C or by the kill command? I am afraid that these methods may cause inconsistency in the file system. Is there a better way?

The system is Ubuntu 16.04. The root partition where /var/log/syslog.1 sits is almost full due to the sudden increase in size of this file as well as /var/log/syslog and /var/log/kern.log. The latter files are still continuing to grow, but the command line is still responsive.

Go to Source
Author: norio

Video4linux: /dev/video0 open failed

I am new to linux and am trying to run the ARToolKit simpleTest program. I am using Ubuntu 14.04.
I try doing ./Configure selecting the video4linux option. When I finally run the ./simpleTest command, it gives me this output:

Using supplied video config string [-dev=dev/video0 ].
video device (dev/video0) open failed

I have already modified the simpleTest.c file as such

#ifdef _WIN32
char            *vconf = "Data\WDM_camera_flipV.xml";
#else
char            *vconf = "-dev=dev/video0 ";
#endif

Please help me understand the problem! My objective is to use the inbuilt webcam of the laptop. Would appreciate any suggestions

Go to Source
Author: pranp2001

Copy and paste a folder

I am trying to make a copy of an OpenFoam tutorial folder to my desktop through Ubuntu since I don’t want to mess up the original files. I use the command line:
cp -r $FOAM_TUTORIAL /mnt/c/Users/username/Desktop

but this error pops up
cp: missing destination file operand after ‘/c/Users/username/Desktop’

Can I get any help with this?

Go to Source
Author: user421564

How to display Apache Tomcat metrics on Ubuntu (Like cPanel’s metrics)

I need a way to display metrics like number of visitors, location of visitors and so on using Ubuntu and Apache Tomcat.
I would like it to work in a cPanel or Webmin fashion where the data is displayed on a webpage and I can access it from another machine (something lightweight too, because my server is a weak machine).

I am using Apache Tomcat v9.0.36 and Ubuntu 18.04.4 LTS.

Thanks

Go to Source
Author: Ivan Dimitrov

how to print tilde symbol on ubuntu

I have environment variable export var=~/.ssh/etc

when I do echo "$var" I get /home/user/.ssh/etc

is there anyway that I can print this variable and get ~/.ssh/etc as an output?

Please Note, that puting it inside single quotes like this export var='~/.ssh/etc' is not an option for me

Go to Source
Author: Kingindanord